10 ms·

Building a personal, private AI computer on a budget

384 points by thm 2y ago

hemant1041 2y ago
Interesting read!
hexomancer 2y ago
Isn't the fact the P40 has horrible fp16 performance a deal breaker for local setups?
- behohippy 2y ago
  You probably won't be running fp16 anything locally. We typically run Q5 or Q6 quants to maximize the size of the model and context length we can run with the VRAM we have available. The quality loss is negligable at Q6.
  - Eisenstein 2y ago
    But the inference doesn't necessarily run at the quant precision.
    - wkat4242 2y ago
      As far as I understand it does if you quantify the K/V store as well (the context). And that's pretty standard now because it can increase maximum context size a lot.
      - Eisenstein 2y ago
        It is available in most inference engines, but I wouldn't call it in standard use, as it can degrade quality tremendously.
        wkat4242 2y ago
        Even at q8_0? I thought it wasn't bad just like the models itself. But very interested to hear. And q8_0 already halves the memory usage compared to fp16. One of the ollama Devs called the quality impact negligible at q8_0: https://smcleod.net/2024/12/bringing-k/v-context-quantisation-to-ollama/ https://smcleod.net/2024/12/bringing-k/v-context-quantisatio... But perhaps quantifying the KV cache does not scale as gracefully as the model itself?
        Eisenstein 2y ago
        It highly depends on the model and the context use. A model like command-r for instance is practically unaffected by it, but Qwen will go nuts. As well, tasks highly dependent on context like translation or evaluation will be more impacted than say, code generation or creative output.
        wkat4242 2y ago
        Interesting. I didn't know that. I thought it was basically 'free' space saving. Would you know how llama3.1 fares by any chance?
        behohippy 2y ago
        Qwen is a little fussy about the sampler settings, but it does run well quantized. If you were getting infinite repetition loops, try dropping the top_p a bit. I think qwen likes lower temps too
        Eisenstein 2y ago
        We are talking about dynamically quantizing KV cache, not the model weights.
        behohippy 2y ago
        I run the KV cache at Q8 even on that model. Is it not working well for you?
- numpad0 2y ago
  Is it cheaper in $/GB than used Vega 56(HBM2 8GB) besides? There are mining boards with bunch of x1 slots that probably can run half a dozen of them for same 48GB.
  - BoredPositron 2y ago
    Would take a bunch of time just to load the model...
  - magicalhippo 2y ago
    AFAIK this doesn't really work for interactive use, as LLMs process data serially. So your request needs to pass through all of the cards for each token, one at a time. Thus a lot of PCIe traffic and hence latency. Better than nothing, but only really useful if you can batch requests so you can keep each GPU working all the time, rather than just one at a time.
    - numpad0 2y ago
      Clearly I wasn't aware enough that DNN is by default like a mesh. Makes sense that it's going to be bottlenecked by the tightest link. Thanks...
- Havoc 2y ago
  Best as I can tell most of the disadvantages relate to larger batches. And for home use you’re likely running batch of 1 anyway
reacharavindh 2y ago
The thing is though.... the locally hosted models in such hardware are cute as toys, and sure do write funny jokes and importantly, perform private tasks that I would never consider passing to non-selfhosted models, but pale in comparison to the models accessible over APIs(Claude 3.5 Sonnet, OpenAI etc). If I could run deepseek-r1-678b locally, without breaking the bank, I would. But, for now, opex > capex at a consumer level.
- walterbell 2y ago
  200+ comments, https://news.ycombinator.com/item?id=42897205 https://news.ycombinator.com/item?id=42897205 > This runs the 671B model in Q4 quantization at 3.5-4.25 TPS for $2K on a single socket Epyc server motherboard using 512GB of RAM.
  - elorant 2y ago
    Runs is an overstatement though. With 4 tokens/second you can't use it on production.
    - deadbabe 2y ago
      Isn’t 4 tps good enough for local use by a single user, which is the point of a personal AI computer?
      - ErikBjare 2y ago
        I tend to get impatient at less than 10tok/s: If the answer is 600tok (normal for me) that's a minute.
      - JKCalhoun 2y ago
        It is for me. I'm happy to switch over to another task (and maybe that task is refilling my coffee) and come back when the answer is fully formed.
      - IanCal 2y ago
        4 tokens per second is pretty slow. That's 5-10s for a comment the length of yours (and R1 specifically likes to output a lot of tokens). It's 10-20x slower than many top end models, which are available cheaply. Even high cost versions of R1 (at more than twice the price of sonnet) are $7/million tokens. For $2K you get 285 million tokens. You'd have to run the box at full whack for over two years (for 4tps) to hit that spending, and that ignores electricity prices. Sonnet 3.5 is half that price, and other R1 providers you could probably hit about a billion tokens for $2k. Gemini flash 2 is over 100 tokens per second and $2k gets you something like 5+B tokens (more really but I'm taking an easy estimate over the more expensive part). If there are things you cannot send to a random party, you might want to look at hosted versions with agreements (if it's a code issue, if you're fine with github then azure is probably fine too). Outside of that, if you really need to then sure, but these are the kinds of things that really benefit from being able to get high usage on GPUs for short periods of time.
dgrabla 2y ago
Great breakdown!. The "own your own AI" at home is a terrific hobby if you like to tinker, but you are going to spend a ton of time and money on hardware that will be underutilized most of the time. If you want to go nuts check out Mitko Vasilev's dream machine. It makes no sense if you don't have a very clear use case that only requires small models or really slow token generation speeds. If the goal however is not to tinker but to really build and learn AI, it is going to be financially better to rent those GPUs/TPUs as needs arise.
- memhole 2y ago
  This is correct. The cost makes no sense outside of hobby and interest. You're far better off renting. I think there is some merit to having a local inference server if you're doing development. You can manage models and have a little more control over your infra as the main benefits.
- theshrike79 2y ago
  Any M-series Mac is "good enough" for home LLMs. Just grab LM studio and a model that fits in memory. Yes, it will not rival OpenAI, but it's 100% local with no monthly fees and depending on the model no censoring or limits on what you can do with it.
- lioeters 2y ago
  > spend a ton of time and money Not necessarily. For non-professional purposes, I've spent zero dollars (no additional memory or GPU) and I'm running a local language model that's good enough to help with many kinds of tasks including writing, coding, and translation. It's a personal, private, budget AI that requires no network connection or third-party servers.
  - ImPostingOnHN 2y ago
    on what hardware (and how much did you spend on it)?
- JKCalhoun 2y ago
  Terrific hobby? Sign me up!
rcarmo 2y ago
Given the power and noise involved, a Mac Mini M4 seems like a much nicer approach, although the RAM requirements will drive up the price.
- deleted 2y ago
  [deleted]
axegon_ 2y ago
I did something similar but using a K80 and M40 I dug up from eBay for pennies. Be advised though, stay as far away as possible from the K80 - the drivers were one of the most painful tech things I've ever had to endure, even if 24GB of VRAM for 50 bucks sounds incredibly appealing. That said, I had a decent-ish HP workstation laying around with 1200 watt power supply so I had where to put those two in. The one thing to note here is that these types of GPUs do not have a cooling of their own. My solution was to 3d print a bunch of brackets and attach several Noctua fans and have them blow at full speed 24/7. Surprisingly it worked way better than I expected - I've never gone above 60 degrees. As a side efffect, the CPUs are also benefiting from this hack: at idle, they are in the mid-20 degrees range. Mind you, the noctua fans are located on the front and the back of the case: the ones on the front act as an intake and the ones on the back as exhaust and there's two more inside the case that are stuck in front of the GPUs. The workstation was refurbished for just over 600 bucks, and another 120 bucks for the GPUs and another ~60 for the fans. Edit: and before someone asks - no I have not uploaded the STL's anywhere cause I haven't had the time but also since this is a very niche use case, though I might: the back(exhaust) bracket came out brilliant the first try - it was a sub-millimeter fit. Then I got cocky and thought that I'd also nail it first try on the intake and ended up re-printing it 4 times.
- JKCalhoun 2y ago
  Curious what HP workstation you have?
  - 9front 2y ago
    HP Z440, it's in the article.
    - JKCalhoun 2y ago
      My comment was not directed at the blog but at the person I responded to.
- egorfine 2y ago
  > K80 - the drivers were one of the most painful tech things I've ever had to endure Well, for a dedicated LLM box it might be feasible to suffer with drivers a bit, no? What was your experience like with the software side?
apples_oranges 2y ago
Does using 2x24GB VRAM mean that the model can be fully loaded into memory if it's between 24 and 48 GB in size? I somehow doubt it, at least ollama wouldn't work like that I think. But does anyone know?
- memhole 2y ago
  No. Hopefully, someone with more knowledge can explain better. But you need room for the kvcache is my understanding. You also need to factor in the size of the context window. If anyone has good resources on this, that would be awesome. Presently, it feels very much like a dark art to host these without crashing or being massively over-provisioned.
  - htrp 2y ago
    The dark art is to massively overprovision hardware.
    - cratermoon 2y ago
      Thus you get Open AI spending billions while DeepSeek comes long with, shocking, actual understanding of the hardware and how to optimize for it[1] and spends $6 Million[2] 1. https://arxiv.org/abs/2412.19437v1 https://arxiv.org/abs/2412.19437v1 2. Quibble over the exact figure. Far less than Open AI, doing more with less.
- Eisenstein 2y ago
  No, you need to have extra space for the context (which requires more space the larger the model is). But it should be said that basing model quality on its size in GB is like qualifying a video based on its size in GB. You can have the same video be small or huge with anywhere from negligible to huge differences in quality between the two. You will be running quantizied model weights, which can range in precision from 1 to 16 bits per parameter (the B for billion in the model name). Model weights at Q8 are generally their parameter size without the B in GB (Llama 3 8B at Q8 would be ~8GB). There are many different strategies for quantizing as well, so this is just a rough guide. So basically if you can't fit the 48GB model into your 48GB of VRAM, just download a lower precision quant.
brador 2y ago
You can do 8b local on the latest iPhones.
- 0xEF 2y ago
  How useful is this, though? In my modest experience, these tiny models aren't good for much more than tinkering with, definitely not something I'd integrate into my workflow since the output quality is pretty low. Again, though, my experience is limited. I imagine others know something I do not and would absolutely love to hear more from people who are running tiny models on low-end hardware for things like code assistance, since that's where the use-case would lie for me. At the moment, I subscribe to "cloud" models that I use for various tasks and that seems to be working well enough, but it would be nice to have a personal model that I could train on very specific data. I'm sure I am missing something, since it's also hard to keep up with all the developments in the Generative AI world.
  - DemetriousJones 2y ago
    I tried running the 8B model on my 8GB M2 Macbook Air through Ollama and it was awful. It took ages to do anything and the responses were bad at best.
    - redman25 2y ago
      Doesn't 8B need at least 16gb of ram? Otherwise, your swapping I would imagine...
      - blebo 2y ago
        Depends on quantization selected - see https://www.canirunthisllm.net/ https://www.canirunthisllm.net/
jmyeet 2y ago
The author mentions it but I want to expand on it: Apple is a seriously good option here, specifically the M4 Mac Mini. What makes Apple attractive is (as the author mentions) that RAM is shared between main and video RAM whereas NVidia is quite intentionally segmenting the market and charging huge premiums for high VRAM cards. Here are some options: 1. Base $599 Mac Mini: 16GB of RAM. Stocked in store. 2. $999 Mac Mini: 24GB of RAM. Stocked in store. 3. Add RAM to either of the above up to 32GB. It's not cheap at $200/8GB but you can buy a Mac Mini with 32GB of shared RAM for $999, substantially cheaper than the author's PC build but less storage (although you can upgrade that too). 4. M4 Pro: $1399 w/ 24GB of RAM. Stocked in store. You can customize this all the way to 64GB of RAM for +$600 so $1999 in total. That is amazing value for this kind of workload. 5. The Mac Studio is really the ultimate option. Way more cores and you can go all the way to 192GB of unified memory (for a $6000 machine). The problem here is that the Mac Studio is old, still on the M2 architecture. An M4 Ultra update is expected sometime this year, possibly late this year. 6. You can get into clustering these (eg [1]). 7. There are various Macbook Pro options, the highest of which is a 16" Mackbook Pro with 128GB of unified memory for $4999. But the main takeaway is the M4 Mac Mini is fantastic value. Some more random thoughts: - Some Mac Minis have Thunderbolt 5 ("TB5"), which is up to either 80Gbps or 120Gbps bidirectional (I've seen it quoted as both); - Mac Minis have the option of 10GbE (+$200); - The Mac Mini has 2 USB3 ports and either 3 TB4 or 3 TB5 ports. [1]: https://blog.exolabs.net/day-2/ https://blog.exolabs.net/day-2/
- sofixa 2y ago
  The issue with Macs is that below Max/Ultra processors, the memory bandwidth is pretty slow. So you need to spend a lot on a high level processor and lots of memory, and the current gen processor, M4, doesn't even have an Ultra, while the Max is only available in a laptop form factor (so thermal constraints). An M4 Pro still has only 273GB/s, while even the 2 generations old RTX 3090 has 935GB/s. https://github.com/ggerganov/llama.cpp/discussions/4167 https://github.com/ggerganov/llama.cpp/discussions/4167
  - jmyeet 2y ago
    That's a good point. I checked the M2 Mac Studio and it's 400GB/s for the M2 Max and 800GB/s for the M2 Ultra so the M4 Ultra when we get it later this year should really be a beast. Oh and the top end Macbook Pro 16 (the only current Mac with an M4 Max) has 410GB/s memory bandwidth. Obviously the Mac Studio is at a much higher price point. Still, you need to spend $1500+ to get an NVidia GPU with >12GB of RAM. Multiple of those starts adding up quick. Put multiple in the same box and you're talking more expensive case, PSU, mainboard, etc and cooling too. Apple has a really interesting opportunity here with their unified memory architecture and power efficiency.
robblbobbl 2y ago
Deep true words. I'm sorry for the author but thx for the article!
gregwebs 2y ago
The problem for me with making such an investment is that next month a better model will be released. It will either require more or less RAM than the current best model- making it either not runnable or expensive to run on an overbuilt machine. Using cloud infrastructure should help with this issue. It may cost much more per run but money can be saved if usage is intermittent. How are HN users handling this?
- nickthegreek 2y ago
  I plan to wait for the NVIDIA Digits release and see what the token/sec is there. Ideally it will work well for at least 2-3 years then I can resell and upgrade if needed.
- diggan 2y ago
  > How are HN users handling this? Combine the best of both worlds. I have a local assistant (communicate via Telegram) that handles tool-calling and basic calendar/todo management (running on a RTX 3090ti), but for more complicated stuff, it can call out to more advanced models (currently using OpenAI APIs for this) granted the request itself doesn't involve personal data, then it flat out refuses, for better or worse.
- idrathernot 2y ago
  There is also an overlooked “tail risk” with cloud services that can end up costing you more than a a few entire on-premise rigs if you don’t correctly configure services or forget to shut down a high end vm instance. Yeah you can implement additional scripts and services as a fail-safe, but this adds another layer of complexity that isn’t always trivial (especially for a hobbyist). I’m not saying that dumping $10k into rapidly depreciating local hardware is the more economical choice, just that people often discount the likelihood and cost of making mistakes in the cloud during their evaluations and the time investment required to ensure you have the correct safeguards in-place.
  - anon373839 2y ago
    Yes. And somehow, those cloud providers just can’t seem to work out how to build a spend limit feature for customers who’d like to prevent that. It must be a really difficult engineering problem…
joshstrange 2y ago
I’d really love to build a machine for local LLMs. I’ve tested models on my MBP M3 Max with 128GB of ram and it’s really cool but I’d like a dedicated local server. I’d also like an excuse to play with proxmox as I’ve just run raw Linux servers or UnRaid w/ containers in the past. I have OpenWebUI and LibreChat running on my local “app server” and I’m quite enjoying that but every time I price out a beefier box I feel like the ROI just isn’t there, especially for an industry that is moving so fast. Privacy is not something to ignore at all but the cost of inference online is very hard to beat, especially when I’m still learning how best to use LLMs.
- datadrivenangel 2y ago
  You pay a premium to get the theoretical local privacy and reliability of hosting your own models. But to get commercially competitive models you need 5 figures of hardware, and then need to actually run it securely and reliably. Pay as you go with multiple vendors as fallback is a better option right now if you don't need harder privacy.
  - rsanek 2y ago
    With something like OpenRouter, you don't even have to manually integrate with multiple vendors
    - wkat4242 2y ago
      Is that like LiteLLM? I have that running but never tried OpenRouter. I wonder now if it's better :)
      - pelatimtt 2y ago
        You can also try LangDB or Portkey
  - joshstrange 2y ago
    Yeah, really I'd love for my Home Assistant to be able to use a local LLM/TTS/STT which I did get working but was way too slow. Also it would fun to just throw some problems/ideas at the wall without incurring (more) cost, that's a big part of it. But each time I run the numbers I would be better off using Anthropic/OpenAI/DeepSeek/other. I think sooner or later I'll break down and buy a server for local inference even if the ROI is upside down because it would be a fun project. I also find that these thing fall in the "You don't know what you will do with it until you have it and it starts unlocking things in your mind"-category. I'm sure there are things I would have it grind on overnight just to test/play with an idea which is something I'd be less likely to do on a paid API.
walterbell 2y ago
One reason to bother with private AI: cloud AI ToS for consumers may have legal clauses about usage of prompt and context data, e.g. data that is not already on the Internet. Enterprise customers can exclude their data from future training. https://stratechery.com/2025/deep-research-and-knowledge-value/ https://stratechery.com/2025/deep-research-and-knowledge-val... > Unless, of course, the information that matters is not on the Internet. This is why I am not sharing the Deep Research report that provoked this insight: I happen to know some things about the industry in question — which is not related to tech, to be clear — because I have a friend who works in it, and it is suddenly clear to me how much future economic value is wrapped up in information not being public. In this case the entity in question is privately held, so there aren’t stock market filings, public reports, barely even a webpage! And so AI is blind. (edited for clarity)
- icepat 2y ago
  ToS can change. Companies can (and do) act illegally. Data breaches happen. Insider threats happen. Why trust the good will of a company, over a box that you built yourself, and have complete control over?
- rovr138 2y ago
  Cost for one
dandanua 2y ago
I doubt it is that efficient. Even though it has 48GB of VRAM, it's more than twice slower than a single 3090 GPU. In my budget AI setup I use 7840 Ryzen based miniPC with USB4 port and connect 3090 to it via the eGPU adapter (ADT-link UT3G). It costed me about $1000 total and I can easily achieve 35 t/s with qwen2.5-coder-32b using ollama.
- mrbonner 2y ago
  Wouldn't eGPU defeat the purpose of having fast memory bandwidth? Have you tried it with stable diffusion?
  - dandanua 2y ago
    40Gbps of USB4 is plenty. I've tried this pytorch tests https://github.com/aime-team/pytorch-benchmarks/ https://github.com/aime-team/pytorch-benchmarks/ and saw only 10% drop in performance. No drop in performance for LLM inference, if a model is already loaded to the VRAM.
    - mrbonner 2y ago
      Wow that makes sense now if you can load the entire model to vram. What eGPU dock and GPU setup you use if you don’t mind?
      - dandanua 2y ago
        That ADT-link UT3G is the eGPU doc, you just need an ordinary PSU to power it and the GPU (I use an old one that I have). Though, to make everything work you might need time to fix quirks. E. g., I had to use "nvidia error 43 fixer" on Windows and to find a correct configuration on my Linux NixOS system (you need to load the Nvidia driver into the kernel in the boot process etc.). Here is how it looks https://imgur.com/a/qySDN4n https://imgur.com/a/qySDN4n
rjurney 2y ago
A lot of people build personal deep learning machines. The economics and convenience can definitely work out... I am confused however by "dummy GPU" - I searched for "dummy" for an explanation but didn't find one. Modern motherboards all include an integrated video card, so I'm not sure what this would be for? My personal DL machine has a 24 core CPU, 128GB RAM and 2 x 3060 GPUs and 2 x 2TB NVMe drives in a RAID 1 array. I <3 it.
- T-A 2y ago
  Look under "Available Graphics" at https://www.hp.com/us-en/shop/mdp/business-solutions/z440-workstation https://www.hp.com/us-en/shop/mdp/business-solutions/z440-wo... No integrated graphics. Author's explanation of the problem: The Teslas are intended to crunch numbers, not to play video games with. Consequently, they don't have any ports to connect a monitor to. The BIOS of the HP Z440 does not like this. It refuses to boot if there is no way to output a video signal.
  - rjurney 2y ago
    Okay, wow I've never heard of one of those!
refibrillator 2y ago
Pay attention to IO bandwidth if you’re building a machine with multiple GPUs like this! In this setup the model is sharded between cards so data must be shuffled through a PCIe 3.0 x16 link which is limited to ~16 GB/s max. For reference that’s an order of magnitude lower than the ~350 GB/s memory bandwidth of the Tesla P40 cards being used. Author didn’t mention NVLink so I’m presuming it wasn’t used, but I believe these cards would support it. Building on a budget is really hard. In my experience 5-15 tok/s is a bit too slow for use cases like coding, but I admit once you’ve had a taste of 150 tok/s it’s hard to go back (I’ve been spoiled by RTX 4090 with vLLM).
- zinccat 2y ago
  I feel that you are mistaking the two bandwidth numbers
- Miraste 2y ago
  Unless you run the GPUs in parallel, which you have to go out of your way to do, the IO bandwidth doesn't matter. The cards hold separate layers of the model, they're not working together. They're only passing a few kilobytes per second between them.
- Xenograph 2y ago
  Which models do you enjoy most on your 4090? and why vLLM instead of ollama?
- ekianjo 2y ago
  > Author didn’t mention NVLink so I’m presuming it wasn’t used, but I believe these cards would support it. How would you setup NVLink, if the cards support it?
black_13 2y ago
[dead]
kamranjon 2y ago
For the same price ($1799) you could buy a Mac Mini with 48gb of unified memory and an m4 pro. It’d probably use less power and be much quieter to run and likely could outperform this setup in terms of tokens per second. I enjoyed the write up still, but I would probably just buy a Mac in this situation.
- diggan 2y ago
  > likely could outperform this setup in terms of tokens per second I've heard arguments both for and against this, but they always lack concrete numbers. I'd love something like "Here is Qwen2.5 at Q4 quantization running via Ollama + these settings, and M4 24GB RAM gets X tokens/s while RTX 3090ti gets Y tokens/s", otherwise we're just propagating mostly anecdotes without any reality-checks.
  - cruffle_duffle 2y ago
    I think we are somewhat still at the “fuzzy super early adopter” stage of this local LLM game and hard data is not going to be easy to come by. I almost want to use the word “hobbiest stage” where almost all of the “data” and “best practice” is anecdotal but I think we are a step above that. Still, it’s way to early and there are simply way to many hardware and software combinations that change almost weekly to establish “the best practice hardware configuration for training / inferencing large language models locally”. Some day there will be established guides with solid. In fact someday there will be be PC’s that specifically target LLMs and will feature all kinds of stats aimed at getting you to bust out your wallet. And I even predict they’ll come up with metrics that all the players will chase well beyond when those metrics make sense (megapixels, clock frequency, etc)… but we aren’t there yet!
    - diggan 2y ago
      Right, but how are we supposed to be getting anywhere else unless people start being more specific and stop leaning on anecdotes or repeating what they've heard elsewhere? Saying "Apple seems to be somewhat equal to this other setup" doesn't really contribute to someone getting an accurate picture if it is equal or not, unless we start including raw numbers, even if they aren't directly comparable. I don't think it's too early to say "I get X tokens/second with this setup + these settings" because then we can at least start comparing, instead of just guessing which seems to be the current SOTA.
zinccat 2y ago
P40 don't support fp16 well, buy 3090 instead
- JKCalhoun 2y ago
  Cost is about 4X for the 24 GB 3090 on eBay.
deleted 2y ago
[deleted]
gwern 2y ago
> Another important finding: Terry is by far the most popular name for a tortoise, followed by Turbo and Toby. Harry is a favorite for hares. All LLMs are loving alliteration. Mode-collapse. One reason that the tuned (or tuning-contaminated models) are bad for creative writing: every protagonist and place seems to be named the same thing.
- diggan 2y ago
  Couldn't you just up the temperature/change some other parameter to get it to be more random/"creative"? It wouldn't be active/intentional randomness/novelty like what a human would do, but at least it shouldn't generate exactly the same naming.
  - gwern 2y ago
    No. The collapse is not as simple as simply shifting down most of the logits, so ramping up the temperature does little until outputs start degenerating.
brianzelip 2y ago
Useful recent podcast about homelab LLMs, https://changelog.com/friends/79 https://changelog.com/friends/79
DrPhish 2y ago
This is just a limited recreation of the ancient mikubox from https://rentry.org/lmg-build-guides https://rentry.org/lmg-build-guides Its funny to see people independently "discover" these builds that are a year plus old. Everyone is sleeping on these guides, but I guess the stink of 4chan scares people away?
- cratermoon 2y ago
  "ancient" guide. Pub: 10 May 2024 21:48 UTC
  - DrPhish 2y ago
    The build guide index page is newer, but to be fair, the mikubox rentry is from Oct 6, 2023. If that isn't "ancient" in terms of AI workstation build guides, then I don't know what is.
miniwark 2y ago
2 x Nvidia Tesla P40 card for €660 is not a thing i consider to be "on a budget". People can play with "small" or "medium" models less powerfull and cheaper cards. A Nvidia Geforce RTX 3060 card with "only" 12Gb VRAM can be found around €200-250 on second hand market (and they are around 300~350 new). In my opinion, 48Gb of VRAM is overkill to call it "on a budget", for me this setup is nice but it's for semi-professional or professional usage. There is of course a trade off to use medium or small models, but being "on a budget" is also to do trade off.
- whywhywhywhy 2y ago
  > A Nvidia Geforce RTX 3060 card with "only" 12Gb VRAM can be found around €200-250 on second hand market 1080Ti might even be a better option, it also has a 12gb model and some reports say it even outperforms the 3060, in non-rtx I presume.
  - Eisenstein 2y ago
    CUDA compute version is a big deal. 1080ti is 6.1. 3060 is 8.6. It also has tensor cores. Note that CUDA version numbers are confusing, the compute number is a different thing than the runtime/driver version.
  - Melatonic 2y ago
    Not sure what used prices are like these days but the Titan XP (similar to the 1080 ti) is even better
- mock-possum 2y ago
  Yeesh, yeah, that was my first thought too - who’s budget?? less than $500 total feels more fitting as a ‘budget’ build - €1700 is more along the lines of ‘enthusiast’ or less charitably “I am rich enough to afford expensive hobbies” If it’s your business and you expect to recoup the cost and write off the cost on your taxes, that’s one thing - but if you’re just looking to run a personal local LLM for funnies, that’s not an accessible price tag. I suppose “or you could just buy a Mac” should have tipped me off though.
cratermoon 2y ago
From the article: In the future, I fully expect to be able to have a frank and honest discussion about the Tiananmen events with an American AI agent, but the only one I can afford will have assumed the persona of Father Christmas who, while holding a can of Coca-Cola, will intersperse the recounting of the tragic events with a joyful "Ho ho ho... Didn't you know? The holidays are coming!" How unfortunate that people are discounting the likelihood that American AI agents will avoid saying things their master think should not be said. Anyone want to take bets on when the big 3 (Open AI, Meta, and Google) will quietly remove anything to do with DEI, trans people, or global warming? They'll start out changing all mentions of "Gulf of Mexico" to "Gulf of America", but then what?
- CamperBob2 2y ago
  It is easy to get a local R1 model to talk about Tiananmen Square to your heart's content. Telling it to replace problematic terms with "Smurf" or another nonsense word is very effective, but with the local model you don't even have to do that in many cases. (e.g., https://i.imgur.com/btcI1fN.png https://i.imgur.com/btcI1fN.png)
  - cratermoon 2y ago
    Ah yes, evidence of Shadow Prompting[1]. Likely their API is given additional, hidden, prompt context making Tiananmen verboten. 1 https://www.techpolicy.press/shining-a-light-on-shadow-prompting/ https://www.techpolicy.press/shining-a-light-on-shadow-promp...
birktj 2y ago
I was wondering if anyone here has experimented with running a cluster of SBC for LLM inference? Ex. the Radxa ROCK 5C has 32GB of memory and also a NPU and only costs about 300 euros. I'm not super up to date on the architecture on modern LLMs, but as far as I understand you should be able to split the layers between multiple nodes? It is not that much data the needs to be sent between them, right? I guess you won't get quite the same performance as a modern mac or nvidia GPU, but it could be quite acceptable and possibly a cheap way of getting a lot of memory. On the other hand I am wondering about what is the state of the art in CPU + GPU inference. Prompt processing is both compute and memory constrained, but I think token generation afterwards is mostly memory bound. Are there any tools that support loading a few layers at a time into a GPU for initial prompt processing and then switches to CPU inference for token generation? Last time I experimented it was possible to run some layers on the GPU and some on the CPU, but to me it seems more efficient to run everything on the GPU initially (but a few layers at a time so they fit in VRAM) and then switch to the CPU when doing the memory bound token generation.
- Eisenstein 2y ago
  > I was wondering if anyone here has experimented with running a cluster of SBC for LLM inference? Ex. the Radxa ROCK 5C has 32GB of memory and also a NPU and only costs about 300 euros. Look into RPC. Llama.cpp supports it. * https://www.reddit.com/r/LocalLLaMA/comments/1cyzi9e/llamacpp_now_supports_distributed_inference/ https://www.reddit.com/r/LocalLLaMA/comments/1cyzi9e/llamacp... > Last time I experimented it was possible to run some layers on the GPU and some on the CPU, but to me it seems more efficient to run everything on the GPU initially (but a few layers at a time so they fit in VRAM) and then switch to the CPU when doing the memory bound token generation. Moving layers over the PCIe bus to do this is going to be slow, which seems to be the issue with that strategy. I think it the key is to use MoE and be smart about which layers go where. This project seems to be doing that with great results: * https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/DeepseekR1_V3_tutorial.md https://github.com/kvcache-ai/ktransformers/blob/main/doc/en...
ollybee 2y ago
The middle ground is to rent a GPU VPS as needed. You can get an H100 for $2/h. Not quite the same privacy as fully local offline, but better than a SASS API and good enough for me. Hopefully in a year or three it will truly be cost effective to run something useful locally and then I can switch.
- 1shooner 2y ago
  Do you have a recommended provider or other pointers for GPU rental?
- anonzzzies 2y ago
  That is what I do but it costs a lot of $, more than just using openrouter. I would like to have a machine so I can have a model talk to itself 24/7 for a realtively fixed price. I have enough solar and wind + cheap net electric so it would basically be free after buying. Just hard to pick what to buy without just forking out a fortune on GPU's.
renewiltord 2y ago
I have 6x 4090s in a rack with an Epyc driving them but tbh I am selling them all to get a Mac Studio. Simpler to work with I think.
_boffin_ 2y ago
For around 1800, I was able to get myself a Dell T5820 with 2x Dell 3090s. Can’t complain at all.
almosthere 2y ago
I bought a Mac M4 Mini (the cheapest one) at Costco for 559 and while I don't know exactly how many tokens per second, it seems to generate text from llama 3.2 (through ollama) as fast as chatgpt.
cwoolfe 2y ago
As others have said, a high powered Mac could be used for the same purpose at a comparable price and lower power usage. Which makes me wonder: why doesn't Apple get into the enterprise AI chip game and compete with Nvidia? They could design their own ASIC for it with all their hardware & manufacturing knowledge. Maybe they already are.
- gmueckl 2y ago
  The primary market for such a product would be businesses. And Apple isn't particularly good at selling to companies. The consumer product focus may just be too ingrained to be successful with such a move. A beefed up home pod with a local LLM-based assistant would be a more typical Apple product. But they'd probably need LLMs to become much, much more reliable to not ruin their reputation over this.
  - fragmede 2y ago
    Why? Siri's still total crap but that doesn't seem to have slowed down iPhone sales.
    - gmueckl 2y ago
      Siri mostly hit the expectations they themselves were able to set through their ads when launching that product - having a voice based assistant at all was huge back then. With an LLM-based assistant, the market has set the expectations for them and they are just unreasonably high and don't mirror reality. That's a potentially big trap for Apple now.
  - lolinder 2y ago
    > And Apple isn't particularly good at selling to companies. With a big glaring exception: developer laptops are overwhelmingly Apple's game right now. It seems like they should be able to piggyback off of that, given that the decision makers are going to be in the same branch of the customer company.
- jrm4 2y ago
  For roughly the same reason Steve Jobs et al killed Hypercard; too much power to the users.
lewisl9029 2y ago
This article is coming out at an interesting time for me. We probably have different definitions for "budget", but I just ordered a super janky eGPU setup for my very dated 8th gen Intel NUC, with a m2->pcie adapter, a PSU, and a refurb Intel A770 for about 350 all-in, not bad considering that's about the cost of a proper Thunderbolt eGPU enclosure alone. The overall idea: A770 seems like a really good budget LLM GPU since it has more memory (16GB) and more memory bandwidth (512GB/s) than a 4070, but costs a tiny fraction. The m2-pcie adapter should give it a bit more bandwidth to the rest of the system than Thunderbolt as well, so hopefully it'll make for a decent gaming experience too. If the eGPU part of the setup doesn't work out for some reason, I'll probably just bite the bullet and order the rest of the PC for a couple hundred more, and return the m2-pcie adapter (I got it off of Amazon instead of Aliexpress specifically so I could do this), looking to end up somewhere around 600 bux total. I think that's probably a more reasonable price of entry for something like this for most people. Curious if anyone else has experience with the A770 for LLM? Been looking at Intel's https://github.com/intel/ipex-llm https://github.com/intel/ipex-llm project and it looked pretty promising, that's what made me pull the trigger in the end. Am I making a huge mistake?
- UncleOxidant 2y ago
  > refurb Intel A770 for about 350 I'm seeing A770s for about $500 - $550. Where did you find a refurb one for $350 (or less since you're also including other parts of the system)
  - lewisl9029 2y ago
    I got this one from Acer's Ebay for $220: https://www.ebay.com/itm/266390922629 https://www.ebay.com/itm/266390922629 It's out of stock now unfortunately, but it does seem to pop up again from time to time according to Slickdeals: https://slickdeals.net/newsearch.php?q=a770&pp=20&sort=newest https://slickdeals.net/newsearch.php?q=a770&pp=20&sort=newes... I would probably just watch the listing and/or set up a deal alert on Slickdeals and wait. If you're in a hurry though, you can probably find a used one on Ebay for not too much more.
jll29 2y ago
Can we just call it "PC to run ML models on" on a budget? "AI computer" sounds pretentious and misleading to outsiders.
whalesalad 2y ago
How many credits does this get you in any cloud or via openai/anthropic api credits? For ~$1700 I can accomplish way more without hardware locally. Don't get me wrong I enjoy tinkering and building projects like this - but it doesn't make financial sense here to me. Unless of course you live 100% off-grid and have Stallman level privacy concerns. Of course I do want my own local GPU compute setup, but the juice just isn't worth the squeeze.
gytisgreitai 2y ago
1.7k€ and 300w for a playground. Man this world is getting crazy and I’m getting f-kin old by not understanding it.
asasidh 2y ago
you can run 32B and even 70B (a bit slow) models on a m4 mac mini pro with 48 GB ram. Out of the box using Ollama. If you enjoy putting together a desktop, thats understandable. https://deepgains.substack.com/p/running-deepseek-locally-for-free https://deepgains.substack.com/p/running-deepseek-locally-fo...
pg5 2y ago
I can run the 17b deepseek models (I know, these smaller ones are not actually deepseek) on my old 1080ti gaming desktop with 64 Gb of RAM. Not exactly speedy, but pretty neat nonetheless.
rootsudo 2y ago
I did something similar and bought a Mac mini. Everyone’s budget is different, right?
windex 2y ago
Ive been running deepseek R1 1.5 on a RPI4. Slow as hell but satisfying.

384 points by thm 2y ago

hemant1041 2y ago

Interesting read!

hexomancer 2y ago

Isn't the fact the P40 has horrible fp16 performance a deal breaker for local setups?

behohippy 2y ago
You probably won't be running fp16 anything locally. We typically run Q5 or Q6 quants to maximize the size of the model and context length we can run with the VRAM we have available. The quality loss is negligable at Q6.
- Eisenstein 2y ago
  But the inference doesn't necessarily run at the quant precision.
  - wkat4242 2y ago
    As far as I understand it does if you quantify the K/V store as well (the context). And that's pretty standard now because it can increase maximum context size a lot.
    - Eisenstein 2y ago
      It is available in most inference engines, but I wouldn't call it in standard use, as it can degrade quality tremendously.
      - wkat4242 2y ago
        Even at q8_0? I thought it wasn't bad just like the models itself. But very interested to hear. And q8_0 already halves the memory usage compared to fp16. One of the ollama Devs called the quality impact negligible at q8_0: https://smcleod.net/2024/12/bringing-k/v-context-quantisation-to-ollama/ https://smcleod.net/2024/12/bringing-k/v-context-quantisatio... But perhaps quantifying the KV cache does not scale as gracefully as the model itself?
        Eisenstein 2y ago
        It highly depends on the model and the context use. A model like command-r for instance is practically unaffected by it, but Qwen will go nuts. As well, tasks highly dependent on context like translation or evaluation will be more impacted than say, code generation or creative output.
        wkat4242 2y ago
        Interesting. I didn't know that. I thought it was basically 'free' space saving. Would you know how llama3.1 fares by any chance?
        behohippy 2y ago
        Qwen is a little fussy about the sampler settings, but it does run well quantized. If you were getting infinite repetition loops, try dropping the top_p a bit. I think qwen likes lower temps too
        Eisenstein 2y ago
        We are talking about dynamically quantizing KV cache, not the model weights.
        behohippy 2y ago
        I run the KV cache at Q8 even on that model. Is it not working well for you?
numpad0 2y ago
Is it cheaper in $/GB than used Vega 56(HBM2 8GB) besides? There are mining boards with bunch of x1 slots that probably can run half a dozen of them for same 48GB.
- BoredPositron 2y ago
  Would take a bunch of time just to load the model...
- magicalhippo 2y ago
  AFAIK this doesn't really work for interactive use, as LLMs process data serially. So your request needs to pass through all of the cards for each token, one at a time. Thus a lot of PCIe traffic and hence latency. Better than nothing, but only really useful if you can batch requests so you can keep each GPU working all the time, rather than just one at a time.
  - numpad0 2y ago
    Clearly I wasn't aware enough that DNN is by default like a mesh. Makes sense that it's going to be bottlenecked by the tightest link. Thanks...
Havoc 2y ago
Best as I can tell most of the disadvantages relate to larger batches. And for home use you’re likely running batch of 1 anyway

reacharavindh 2y ago

The thing is though.... the locally hosted models in such hardware are cute as toys, and sure do write funny jokes and importantly, perform private tasks that I would never consider passing to non-selfhosted models, but pale in comparison to the models accessible over APIs(Claude 3.5 Sonnet, OpenAI etc). If I could run deepseek-r1-678b locally, without breaking the bank, I would. But, for now, opex > capex at a consumer level.

walterbell 2y ago
200+ comments, https://news.ycombinator.com/item?id=42897205 https://news.ycombinator.com/item?id=42897205 > This runs the 671B model in Q4 quantization at 3.5-4.25 TPS for $2K on a single socket Epyc server motherboard using 512GB of RAM.
- elorant 2y ago
  Runs is an overstatement though. With 4 tokens/second you can't use it on production.
  - deadbabe 2y ago
    Isn’t 4 tps good enough for local use by a single user, which is the point of a personal AI computer?
    - ErikBjare 2y ago
      I tend to get impatient at less than 10tok/s: If the answer is 600tok (normal for me) that's a minute.
    - JKCalhoun 2y ago
      It is for me. I'm happy to switch over to another task (and maybe that task is refilling my coffee) and come back when the answer is fully formed.
    - IanCal 2y ago
      4 tokens per second is pretty slow. That's 5-10s for a comment the length of yours (and R1 specifically likes to output a lot of tokens). It's 10-20x slower than many top end models, which are available cheaply. Even high cost versions of R1 (at more than twice the price of sonnet) are $7/million tokens. For $2K you get 285 million tokens. You'd have to run the box at full whack for over two years (for 4tps) to hit that spending, and that ignores electricity prices. Sonnet 3.5 is half that price, and other R1 providers you could probably hit about a billion tokens for $2k. Gemini flash 2 is over 100 tokens per second and $2k gets you something like 5+B tokens (more really but I'm taking an easy estimate over the more expensive part). If there are things you cannot send to a random party, you might want to look at hosted versions with agreements (if it's a code issue, if you're fine with github then azure is probably fine too). Outside of that, if you really need to then sure, but these are the kinds of things that really benefit from being able to get high usage on GPUs for short periods of time.

dgrabla 2y ago

Great breakdown!. The "own your own AI" at home is a terrific hobby if you like to tinker, but you are going to spend a ton of time and money on hardware that will be underutilized most of the time. If you want to go nuts check out Mitko Vasilev's dream machine. It makes no sense if you don't have a very clear use case that only requires small models or really slow token generation speeds. If the goal however is not to tinker but to really build and learn AI, it is going to be financially better to rent those GPUs/TPUs as needs arise.

memhole 2y ago
This is correct. The cost makes no sense outside of hobby and interest. You're far better off renting. I think there is some merit to having a local inference server if you're doing development. You can manage models and have a little more control over your infra as the main benefits.
theshrike79 2y ago
Any M-series Mac is "good enough" for home LLMs. Just grab LM studio and a model that fits in memory. Yes, it will not rival OpenAI, but it's 100% local with no monthly fees and depending on the model no censoring or limits on what you can do with it.
lioeters 2y ago
> spend a ton of time and money Not necessarily. For non-professional purposes, I've spent zero dollars (no additional memory or GPU) and I'm running a local language model that's good enough to help with many kinds of tasks including writing, coding, and translation. It's a personal, private, budget AI that requires no network connection or third-party servers.
- ImPostingOnHN 2y ago
  on what hardware (and how much did you spend on it)?
JKCalhoun 2y ago
Terrific hobby? Sign me up!

rcarmo 2y ago

Given the power and noise involved, a Mac Mini M4 seems like a much nicer approach, although the RAM requirements will drive up the price.

deleted 2y ago
[deleted]

axegon_ 2y ago

I did something similar but using a K80 and M40 I dug up from eBay for pennies. Be advised though, stay as far away as possible from the K80 - the drivers were one of the most painful tech things I've ever had to endure, even if 24GB of VRAM for 50 bucks sounds incredibly appealing. That said, I had a decent-ish HP workstation laying around with 1200 watt power supply so I had where to put those two in. The one thing to note here is that these types of GPUs do not have a cooling of their own. My solution was to 3d print a bunch of brackets and attach several Noctua fans and have them blow at full speed 24/7. Surprisingly it worked way better than I expected - I've never gone above 60 degrees. As a side efffect, the CPUs are also benefiting from this hack: at idle, they are in the mid-20 degrees range. Mind you, the noctua fans are located on the front and the back of the case: the ones on the front act as an intake and the ones on the back as exhaust and there's two more inside the case that are stuck in front of the GPUs. The workstation was refurbished for just over 600 bucks, and another 120 bucks for the GPUs and another ~60 for the fans. Edit: and before someone asks - no I have not uploaded the STL's anywhere cause I haven't had the time but also since this is a very niche use case, though I might: the back(exhaust) bracket came out brilliant the first try - it was a sub-millimeter fit. Then I got cocky and thought that I'd also nail it first try on the intake and ended up re-printing it 4 times.

JKCalhoun 2y ago
Curious what HP workstation you have?
- 9front 2y ago
  HP Z440, it's in the article.
  - JKCalhoun 2y ago
    My comment was not directed at the blog but at the person I responded to.
egorfine 2y ago
> K80 - the drivers were one of the most painful tech things I've ever had to endure Well, for a dedicated LLM box it might be feasible to suffer with drivers a bit, no? What was your experience like with the software side?

apples_oranges 2y ago

Does using 2x24GB VRAM mean that the model can be fully loaded into memory if it's between 24 and 48 GB in size? I somehow doubt it, at least ollama wouldn't work like that I think. But does anyone know?

memhole 2y ago
No. Hopefully, someone with more knowledge can explain better. But you need room for the kvcache is my understanding. You also need to factor in the size of the context window. If anyone has good resources on this, that would be awesome. Presently, it feels very much like a dark art to host these without crashing or being massively over-provisioned.
- htrp 2y ago
  The dark art is to massively overprovision hardware.
  - cratermoon 2y ago
    Thus you get Open AI spending billions while DeepSeek comes long with, shocking, actual understanding of the hardware and how to optimize for it[1] and spends $6 Million[2] 1. https://arxiv.org/abs/2412.19437v1 https://arxiv.org/abs/2412.19437v1 2. Quibble over the exact figure. Far less than Open AI, doing more with less.
Eisenstein 2y ago
No, you need to have extra space for the context (which requires more space the larger the model is). But it should be said that basing model quality on its size in GB is like qualifying a video based on its size in GB. You can have the same video be small or huge with anywhere from negligible to huge differences in quality between the two. You will be running quantizied model weights, which can range in precision from 1 to 16 bits per parameter (the B for billion in the model name). Model weights at Q8 are generally their parameter size without the B in GB (Llama 3 8B at Q8 would be ~8GB). There are many different strategies for quantizing as well, so this is just a rough guide. So basically if you can't fit the 48GB model into your 48GB of VRAM, just download a lower precision quant.

brador 2y ago

You can do 8b local on the latest iPhones.

0xEF 2y ago
How useful is this, though? In my modest experience, these tiny models aren't good for much more than tinkering with, definitely not something I'd integrate into my workflow since the output quality is pretty low. Again, though, my experience is limited. I imagine others know something I do not and would absolutely love to hear more from people who are running tiny models on low-end hardware for things like code assistance, since that's where the use-case would lie for me. At the moment, I subscribe to "cloud" models that I use for various tasks and that seems to be working well enough, but it would be nice to have a personal model that I could train on very specific data. I'm sure I am missing something, since it's also hard to keep up with all the developments in the Generative AI world.
- DemetriousJones 2y ago
  I tried running the 8B model on my 8GB M2 Macbook Air through Ollama and it was awful. It took ages to do anything and the responses were bad at best.
  - redman25 2y ago
    Doesn't 8B need at least 16gb of ram? Otherwise, your swapping I would imagine...
    - blebo 2y ago
      Depends on quantization selected - see https://www.canirunthisllm.net/ https://www.canirunthisllm.net/

jmyeet 2y ago

The author mentions it but I want to expand on it: Apple is a seriously good option here, specifically the M4 Mac Mini. What makes Apple attractive is (as the author mentions) that RAM is shared between main and video RAM whereas NVidia is quite intentionally segmenting the market and charging huge premiums for high VRAM cards. Here are some options: 1. Base $599 Mac Mini: 16GB of RAM. Stocked in store. 2. $999 Mac Mini: 24GB of RAM. Stocked in store. 3. Add RAM to either of the above up to 32GB. It's not cheap at $200/8GB but you can buy a Mac Mini with 32GB of shared RAM for $999, substantially cheaper than the author's PC build but less storage (although you can upgrade that too). 4. M4 Pro: $1399 w/ 24GB of RAM. Stocked in store. You can customize this all the way to 64GB of RAM for +$600 so $1999 in total. That is amazing value for this kind of workload. 5. The Mac Studio is really the ultimate option. Way more cores and you can go all the way to 192GB of unified memory (for a $6000 machine). The problem here is that the Mac Studio is old, still on the M2 architecture. An M4 Ultra update is expected sometime this year, possibly late this year. 6. You can get into clustering these (eg [1]). 7. There are various Macbook Pro options, the highest of which is a 16" Mackbook Pro with 128GB of unified memory for $4999. But the main takeaway is the M4 Mac Mini is fantastic value. Some more random thoughts: - Some Mac Minis have Thunderbolt 5 ("TB5"), which is up to either 80Gbps or 120Gbps bidirectional (I've seen it quoted as both); - Mac Minis have the option of 10GbE (+$200); - The Mac Mini has 2 USB3 ports and either 3 TB4 or 3 TB5 ports. [1]: https://blog.exolabs.net/day-2/ https://blog.exolabs.net/day-2/

sofixa 2y ago
The issue with Macs is that below Max/Ultra processors, the memory bandwidth is pretty slow. So you need to spend a lot on a high level processor and lots of memory, and the current gen processor, M4, doesn't even have an Ultra, while the Max is only available in a laptop form factor (so thermal constraints). An M4 Pro still has only 273GB/s, while even the 2 generations old RTX 3090 has 935GB/s. https://github.com/ggerganov/llama.cpp/discussions/4167 https://github.com/ggerganov/llama.cpp/discussions/4167
- jmyeet 2y ago
  That's a good point. I checked the M2 Mac Studio and it's 400GB/s for the M2 Max and 800GB/s for the M2 Ultra so the M4 Ultra when we get it later this year should really be a beast. Oh and the top end Macbook Pro 16 (the only current Mac with an M4 Max) has 410GB/s memory bandwidth. Obviously the Mac Studio is at a much higher price point. Still, you need to spend $1500+ to get an NVidia GPU with >12GB of RAM. Multiple of those starts adding up quick. Put multiple in the same box and you're talking more expensive case, PSU, mainboard, etc and cooling too. Apple has a really interesting opportunity here with their unified memory architecture and power efficiency.

robblbobbl 2y ago

Deep true words. I'm sorry for the author but thx for the article!

gregwebs 2y ago

The problem for me with making such an investment is that next month a better model will be released. It will either require more or less RAM than the current best model- making it either not runnable or expensive to run on an overbuilt machine. Using cloud infrastructure should help with this issue. It may cost much more per run but money can be saved if usage is intermittent. How are HN users handling this?

nickthegreek 2y ago
I plan to wait for the NVIDIA Digits release and see what the token/sec is there. Ideally it will work well for at least 2-3 years then I can resell and upgrade if needed.
diggan 2y ago
> How are HN users handling this? Combine the best of both worlds. I have a local assistant (communicate via Telegram) that handles tool-calling and basic calendar/todo management (running on a RTX 3090ti), but for more complicated stuff, it can call out to more advanced models (currently using OpenAI APIs for this) granted the request itself doesn't involve personal data, then it flat out refuses, for better or worse.
idrathernot 2y ago
There is also an overlooked “tail risk” with cloud services that can end up costing you more than a a few entire on-premise rigs if you don’t correctly configure services or forget to shut down a high end vm instance. Yeah you can implement additional scripts and services as a fail-safe, but this adds another layer of complexity that isn’t always trivial (especially for a hobbyist). I’m not saying that dumping $10k into rapidly depreciating local hardware is the more economical choice, just that people often discount the likelihood and cost of making mistakes in the cloud during their evaluations and the time investment required to ensure you have the correct safeguards in-place.
- anon373839 2y ago
  Yes. And somehow, those cloud providers just can’t seem to work out how to build a spend limit feature for customers who’d like to prevent that. It must be a really difficult engineering problem…

joshstrange 2y ago

I’d really love to build a machine for local LLMs. I’ve tested models on my MBP M3 Max with 128GB of ram and it’s really cool but I’d like a dedicated local server. I’d also like an excuse to play with proxmox as I’ve just run raw Linux servers or UnRaid w/ containers in the past. I have OpenWebUI and LibreChat running on my local “app server” and I’m quite enjoying that but every time I price out a beefier box I feel like the ROI just isn’t there, especially for an industry that is moving so fast. Privacy is not something to ignore at all but the cost of inference online is very hard to beat, especially when I’m still learning how best to use LLMs.

datadrivenangel 2y ago
You pay a premium to get the theoretical local privacy and reliability of hosting your own models. But to get commercially competitive models you need 5 figures of hardware, and then need to actually run it securely and reliably. Pay as you go with multiple vendors as fallback is a better option right now if you don't need harder privacy.
- rsanek 2y ago
  With something like OpenRouter, you don't even have to manually integrate with multiple vendors
  - wkat4242 2y ago
    Is that like LiteLLM? I have that running but never tried OpenRouter. I wonder now if it's better :)
    - pelatimtt 2y ago
      You can also try LangDB or Portkey
- joshstrange 2y ago
  Yeah, really I'd love for my Home Assistant to be able to use a local LLM/TTS/STT which I did get working but was way too slow. Also it would fun to just throw some problems/ideas at the wall without incurring (more) cost, that's a big part of it. But each time I run the numbers I would be better off using Anthropic/OpenAI/DeepSeek/other. I think sooner or later I'll break down and buy a server for local inference even if the ROI is upside down because it would be a fun project. I also find that these thing fall in the "You don't know what you will do with it until you have it and it starts unlocking things in your mind"-category. I'm sure there are things I would have it grind on overnight just to test/play with an idea which is something I'd be less likely to do on a paid API.

walterbell 2y ago

One reason to bother with private AI: cloud AI ToS for consumers may have legal clauses about usage of prompt and context data, e.g. data that is not already on the Internet. Enterprise customers can exclude their data from future training. https://stratechery.com/2025/deep-research-and-knowledge-value/ https://stratechery.com/2025/deep-research-and-knowledge-val... > Unless, of course, the information that matters is not on the Internet. This is why I am not sharing the Deep Research report that provoked this insight: I happen to know some things about the industry in question — which is not related to tech, to be clear — because I have a friend who works in it, and it is suddenly clear to me how much future economic value is wrapped up in information not being public. In this case the entity in question is privately held, so there aren’t stock market filings, public reports, barely even a webpage! And so AI is blind. (edited for clarity)

icepat 2y ago
ToS can change. Companies can (and do) act illegally. Data breaches happen. Insider threats happen. Why trust the good will of a company, over a box that you built yourself, and have complete control over?
rovr138 2y ago
Cost for one

dandanua 2y ago

I doubt it is that efficient. Even though it has 48GB of VRAM, it's more than twice slower than a single 3090 GPU. In my budget AI setup I use 7840 Ryzen based miniPC with USB4 port and connect 3090 to it via the eGPU adapter (ADT-link UT3G). It costed me about $1000 total and I can easily achieve 35 t/s with qwen2.5-coder-32b using ollama.

mrbonner 2y ago
Wouldn't eGPU defeat the purpose of having fast memory bandwidth? Have you tried it with stable diffusion?
- dandanua 2y ago
  40Gbps of USB4 is plenty. I've tried this pytorch tests https://github.com/aime-team/pytorch-benchmarks/ https://github.com/aime-team/pytorch-benchmarks/ and saw only 10% drop in performance. No drop in performance for LLM inference, if a model is already loaded to the VRAM.
  - mrbonner 2y ago
    Wow that makes sense now if you can load the entire model to vram. What eGPU dock and GPU setup you use if you don’t mind?
    - dandanua 2y ago
      That ADT-link UT3G is the eGPU doc, you just need an ordinary PSU to power it and the GPU (I use an old one that I have). Though, to make everything work you might need time to fix quirks. E. g., I had to use "nvidia error 43 fixer" on Windows and to find a correct configuration on my Linux NixOS system (you need to load the Nvidia driver into the kernel in the boot process etc.). Here is how it looks https://imgur.com/a/qySDN4n https://imgur.com/a/qySDN4n

rjurney 2y ago

A lot of people build personal deep learning machines. The economics and convenience can definitely work out... I am confused however by "dummy GPU" - I searched for "dummy" for an explanation but didn't find one. Modern motherboards all include an integrated video card, so I'm not sure what this would be for? My personal DL machine has a 24 core CPU, 128GB RAM and 2 x 3060 GPUs and 2 x 2TB NVMe drives in a RAID 1 array. I <3 it.

T-A 2y ago
Look under "Available Graphics" at https://www.hp.com/us-en/shop/mdp/business-solutions/z440-workstation https://www.hp.com/us-en/shop/mdp/business-solutions/z440-wo... No integrated graphics. Author's explanation of the problem: The Teslas are intended to crunch numbers, not to play video games with. Consequently, they don't have any ports to connect a monitor to. The BIOS of the HP Z440 does not like this. It refuses to boot if there is no way to output a video signal.
- rjurney 2y ago
  Okay, wow I've never heard of one of those!

refibrillator 2y ago

Pay attention to IO bandwidth if you’re building a machine with multiple GPUs like this! In this setup the model is sharded between cards so data must be shuffled through a PCIe 3.0 x16 link which is limited to ~16 GB/s max. For reference that’s an order of magnitude lower than the ~350 GB/s memory bandwidth of the Tesla P40 cards being used. Author didn’t mention NVLink so I’m presuming it wasn’t used, but I believe these cards would support it. Building on a budget is really hard. In my experience 5-15 tok/s is a bit too slow for use cases like coding, but I admit once you’ve had a taste of 150 tok/s it’s hard to go back (I’ve been spoiled by RTX 4090 with vLLM).

zinccat 2y ago
I feel that you are mistaking the two bandwidth numbers
Miraste 2y ago
Unless you run the GPUs in parallel, which you have to go out of your way to do, the IO bandwidth doesn't matter. The cards hold separate layers of the model, they're not working together. They're only passing a few kilobytes per second between them.
Xenograph 2y ago
Which models do you enjoy most on your 4090? and why vLLM instead of ollama?
ekianjo 2y ago
> Author didn’t mention NVLink so I’m presuming it wasn’t used, but I believe these cards would support it. How would you setup NVLink, if the cards support it?

black_13 2y ago

[dead]

kamranjon 2y ago

For the same price ($1799) you could buy a Mac Mini with 48gb of unified memory and an m4 pro. It’d probably use less power and be much quieter to run and likely could outperform this setup in terms of tokens per second. I enjoyed the write up still, but I would probably just buy a Mac in this situation.

diggan 2y ago
> likely could outperform this setup in terms of tokens per second I've heard arguments both for and against this, but they always lack concrete numbers. I'd love something like "Here is Qwen2.5 at Q4 quantization running via Ollama + these settings, and M4 24GB RAM gets X tokens/s while RTX 3090ti gets Y tokens/s", otherwise we're just propagating mostly anecdotes without any reality-checks.
- cruffle_duffle 2y ago
  I think we are somewhat still at the “fuzzy super early adopter” stage of this local LLM game and hard data is not going to be easy to come by. I almost want to use the word “hobbiest stage” where almost all of the “data” and “best practice” is anecdotal but I think we are a step above that. Still, it’s way to early and there are simply way to many hardware and software combinations that change almost weekly to establish “the best practice hardware configuration for training / inferencing large language models locally”. Some day there will be established guides with solid. In fact someday there will be be PC’s that specifically target LLMs and will feature all kinds of stats aimed at getting you to bust out your wallet. And I even predict they’ll come up with metrics that all the players will chase well beyond when those metrics make sense (megapixels, clock frequency, etc)… but we aren’t there yet!
  - diggan 2y ago
    Right, but how are we supposed to be getting anywhere else unless people start being more specific and stop leaning on anecdotes or repeating what they've heard elsewhere? Saying "Apple seems to be somewhat equal to this other setup" doesn't really contribute to someone getting an accurate picture if it is equal or not, unless we start including raw numbers, even if they aren't directly comparable. I don't think it's too early to say "I get X tokens/second with this setup + these settings" because then we can at least start comparing, instead of just guessing which seems to be the current SOTA.

zinccat 2y ago

P40 don't support fp16 well, buy 3090 instead

JKCalhoun 2y ago
Cost is about 4X for the 24 GB 3090 on eBay.

deleted 2y ago

[deleted]

gwern 2y ago

> Another important finding: Terry is by far the most popular name for a tortoise, followed by Turbo and Toby. Harry is a favorite for hares. All LLMs are loving alliteration. Mode-collapse. One reason that the tuned (or tuning-contaminated models) are bad for creative writing: every protagonist and place seems to be named the same thing.

diggan 2y ago
Couldn't you just up the temperature/change some other parameter to get it to be more random/"creative"? It wouldn't be active/intentional randomness/novelty like what a human would do, but at least it shouldn't generate exactly the same naming.
- gwern 2y ago
  No. The collapse is not as simple as simply shifting down most of the logits, so ramping up the temperature does little until outputs start degenerating.

brianzelip 2y ago

Useful recent podcast about homelab LLMs, https://changelog.com/friends/79 https://changelog.com/friends/79

DrPhish 2y ago

This is just a limited recreation of the ancient mikubox from https://rentry.org/lmg-build-guides https://rentry.org/lmg-build-guides Its funny to see people independently "discover" these builds that are a year plus old. Everyone is sleeping on these guides, but I guess the stink of 4chan scares people away?

cratermoon 2y ago
"ancient" guide. Pub: 10 May 2024 21:48 UTC
- DrPhish 2y ago
  The build guide index page is newer, but to be fair, the mikubox rentry is from Oct 6, 2023. If that isn't "ancient" in terms of AI workstation build guides, then I don't know what is.

miniwark 2y ago

2 x Nvidia Tesla P40 card for €660 is not a thing i consider to be "on a budget". People can play with "small" or "medium" models less powerfull and cheaper cards. A Nvidia Geforce RTX 3060 card with "only" 12Gb VRAM can be found around €200-250 on second hand market (and they are around 300~350 new). In my opinion, 48Gb of VRAM is overkill to call it "on a budget", for me this setup is nice but it's for semi-professional or professional usage. There is of course a trade off to use medium or small models, but being "on a budget" is also to do trade off.

whywhywhywhy 2y ago
> A Nvidia Geforce RTX 3060 card with "only" 12Gb VRAM can be found around €200-250 on second hand market 1080Ti might even be a better option, it also has a 12gb model and some reports say it even outperforms the 3060, in non-rtx I presume.
- Eisenstein 2y ago
  CUDA compute version is a big deal. 1080ti is 6.1. 3060 is 8.6. It also has tensor cores. Note that CUDA version numbers are confusing, the compute number is a different thing than the runtime/driver version.
- Melatonic 2y ago
  Not sure what used prices are like these days but the Titan XP (similar to the 1080 ti) is even better
mock-possum 2y ago
Yeesh, yeah, that was my first thought too - who’s budget?? less than $500 total feels more fitting as a ‘budget’ build - €1700 is more along the lines of ‘enthusiast’ or less charitably “I am rich enough to afford expensive hobbies” If it’s your business and you expect to recoup the cost and write off the cost on your taxes, that’s one thing - but if you’re just looking to run a personal local LLM for funnies, that’s not an accessible price tag. I suppose “or you could just buy a Mac” should have tipped me off though.

cratermoon 2y ago

From the article: In the future, I fully expect to be able to have a frank and honest discussion about the Tiananmen events with an American AI agent, but the only one I can afford will have assumed the persona of Father Christmas who, while holding a can of Coca-Cola, will intersperse the recounting of the tragic events with a joyful "Ho ho ho... Didn't you know? The holidays are coming!" How unfortunate that people are discounting the likelihood that American AI agents will avoid saying things their master think should not be said. Anyone want to take bets on when the big 3 (Open AI, Meta, and Google) will quietly remove anything to do with DEI, trans people, or global warming? They'll start out changing all mentions of "Gulf of Mexico" to "Gulf of America", but then what?

CamperBob2 2y ago
It is easy to get a local R1 model to talk about Tiananmen Square to your heart's content. Telling it to replace problematic terms with "Smurf" or another nonsense word is very effective, but with the local model you don't even have to do that in many cases. (e.g., https://i.imgur.com/btcI1fN.png https://i.imgur.com/btcI1fN.png)
- cratermoon 2y ago
  Ah yes, evidence of Shadow Prompting[1]. Likely their API is given additional, hidden, prompt context making Tiananmen verboten. 1 https://www.techpolicy.press/shining-a-light-on-shadow-prompting/ https://www.techpolicy.press/shining-a-light-on-shadow-promp...

birktj 2y ago

I was wondering if anyone here has experimented with running a cluster of SBC for LLM inference? Ex. the Radxa ROCK 5C has 32GB of memory and also a NPU and only costs about 300 euros. I'm not super up to date on the architecture on modern LLMs, but as far as I understand you should be able to split the layers between multiple nodes? It is not that much data the needs to be sent between them, right? I guess you won't get quite the same performance as a modern mac or nvidia GPU, but it could be quite acceptable and possibly a cheap way of getting a lot of memory. On the other hand I am wondering about what is the state of the art in CPU + GPU inference. Prompt processing is both compute and memory constrained, but I think token generation afterwards is mostly memory bound. Are there any tools that support loading a few layers at a time into a GPU for initial prompt processing and then switches to CPU inference for token generation? Last time I experimented it was possible to run some layers on the GPU and some on the CPU, but to me it seems more efficient to run everything on the GPU initially (but a few layers at a time so they fit in VRAM) and then switch to the CPU when doing the memory bound token generation.

Eisenstein 2y ago
> I was wondering if anyone here has experimented with running a cluster of SBC for LLM inference? Ex. the Radxa ROCK 5C has 32GB of memory and also a NPU and only costs about 300 euros. Look into RPC. Llama.cpp supports it. * https://www.reddit.com/r/LocalLLaMA/comments/1cyzi9e/llamacpp_now_supports_distributed_inference/ https://www.reddit.com/r/LocalLLaMA/comments/1cyzi9e/llamacp... > Last time I experimented it was possible to run some layers on the GPU and some on the CPU, but to me it seems more efficient to run everything on the GPU initially (but a few layers at a time so they fit in VRAM) and then switch to the CPU when doing the memory bound token generation. Moving layers over the PCIe bus to do this is going to be slow, which seems to be the issue with that strategy. I think it the key is to use MoE and be smart about which layers go where. This project seems to be doing that with great results: * https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/DeepseekR1_V3_tutorial.md https://github.com/kvcache-ai/ktransformers/blob/main/doc/en...

ollybee 2y ago

The middle ground is to rent a GPU VPS as needed. You can get an H100 for $2/h. Not quite the same privacy as fully local offline, but better than a SASS API and good enough for me. Hopefully in a year or three it will truly be cost effective to run something useful locally and then I can switch.

1shooner 2y ago
Do you have a recommended provider or other pointers for GPU rental?
anonzzzies 2y ago
That is what I do but it costs a lot of $, more than just using openrouter. I would like to have a machine so I can have a model talk to itself 24/7 for a realtively fixed price. I have enough solar and wind + cheap net electric so it would basically be free after buying. Just hard to pick what to buy without just forking out a fortune on GPU's.

renewiltord 2y ago

I have 6x 4090s in a rack with an Epyc driving them but tbh I am selling them all to get a Mac Studio. Simpler to work with I think.

_boffin_ 2y ago

For around 1800, I was able to get myself a Dell T5820 with 2x Dell 3090s. Can’t complain at all.

almosthere 2y ago

I bought a Mac M4 Mini (the cheapest one) at Costco for 559 and while I don't know exactly how many tokens per second, it seems to generate text from llama 3.2 (through ollama) as fast as chatgpt.

cwoolfe 2y ago

As others have said, a high powered Mac could be used for the same purpose at a comparable price and lower power usage. Which makes me wonder: why doesn't Apple get into the enterprise AI chip game and compete with Nvidia? They could design their own ASIC for it with all their hardware & manufacturing knowledge. Maybe they already are.

gmueckl 2y ago
The primary market for such a product would be businesses. And Apple isn't particularly good at selling to companies. The consumer product focus may just be too ingrained to be successful with such a move. A beefed up home pod with a local LLM-based assistant would be a more typical Apple product. But they'd probably need LLMs to become much, much more reliable to not ruin their reputation over this.
- fragmede 2y ago
  Why? Siri's still total crap but that doesn't seem to have slowed down iPhone sales.
  - gmueckl 2y ago
    Siri mostly hit the expectations they themselves were able to set through their ads when launching that product - having a voice based assistant at all was huge back then. With an LLM-based assistant, the market has set the expectations for them and they are just unreasonably high and don't mirror reality. That's a potentially big trap for Apple now.
- lolinder 2y ago
  > And Apple isn't particularly good at selling to companies. With a big glaring exception: developer laptops are overwhelmingly Apple's game right now. It seems like they should be able to piggyback off of that, given that the decision makers are going to be in the same branch of the customer company.
jrm4 2y ago
For roughly the same reason Steve Jobs et al killed Hypercard; too much power to the users.

lewisl9029 2y ago

This article is coming out at an interesting time for me. We probably have different definitions for "budget", but I just ordered a super janky eGPU setup for my very dated 8th gen Intel NUC, with a m2->pcie adapter, a PSU, and a refurb Intel A770 for about 350 all-in, not bad considering that's about the cost of a proper Thunderbolt eGPU enclosure alone. The overall idea: A770 seems like a really good budget LLM GPU since it has more memory (16GB) and more memory bandwidth (512GB/s) than a 4070, but costs a tiny fraction. The m2-pcie adapter should give it a bit more bandwidth to the rest of the system than Thunderbolt as well, so hopefully it'll make for a decent gaming experience too. If the eGPU part of the setup doesn't work out for some reason, I'll probably just bite the bullet and order the rest of the PC for a couple hundred more, and return the m2-pcie adapter (I got it off of Amazon instead of Aliexpress specifically so I could do this), looking to end up somewhere around 600 bux total. I think that's probably a more reasonable price of entry for something like this for most people. Curious if anyone else has experience with the A770 for LLM? Been looking at Intel's https://github.com/intel/ipex-llm https://github.com/intel/ipex-llm project and it looked pretty promising, that's what made me pull the trigger in the end. Am I making a huge mistake?

UncleOxidant 2y ago
> refurb Intel A770 for about 350 I'm seeing A770s for about $500 - $550. Where did you find a refurb one for $350 (or less since you're also including other parts of the system)
- lewisl9029 2y ago
  I got this one from Acer's Ebay for $220: https://www.ebay.com/itm/266390922629 https://www.ebay.com/itm/266390922629 It's out of stock now unfortunately, but it does seem to pop up again from time to time according to Slickdeals: https://slickdeals.net/newsearch.php?q=a770&pp=20&sort=newest https://slickdeals.net/newsearch.php?q=a770&pp=20&sort=newes... I would probably just watch the listing and/or set up a deal alert on Slickdeals and wait. If you're in a hurry though, you can probably find a used one on Ebay for not too much more.

jll29 2y ago

Can we just call it "PC to run ML models on" on a budget? "AI computer" sounds pretentious and misleading to outsiders.

whalesalad 2y ago

How many credits does this get you in any cloud or via openai/anthropic api credits? For ~$1700 I can accomplish way more without hardware locally. Don't get me wrong I enjoy tinkering and building projects like this - but it doesn't make financial sense here to me. Unless of course you live 100% off-grid and have Stallman level privacy concerns. Of course I do want my own local GPU compute setup, but the juice just isn't worth the squeeze.

gytisgreitai 2y ago

1.7k€ and 300w for a playground. Man this world is getting crazy and I’m getting f-kin old by not understanding it.

asasidh 2y ago

you can run 32B and even 70B (a bit slow) models on a m4 mac mini pro with 48 GB ram. Out of the box using Ollama. If you enjoy putting together a desktop, thats understandable. https://deepgains.substack.com/p/running-deepseek-locally-for-free https://deepgains.substack.com/p/running-deepseek-locally-fo...

pg5 2y ago

I can run the 17b deepseek models (I know, these smaller ones are not actually deepseek) on my old 1080ti gaming desktop with 64 Gb of RAM. Not exactly speedy, but pretty neat nonetheless.

rootsudo 2y ago

I did something similar and bought a Mac mini. Everyone’s budget is different, right?

windex 2y ago

Ive been running deepseek R1 1.5 on a RPI4. Slow as hell but satisfying.