7 ms·

I put a datacenter GPU in my gaming PC

332 points by birdculture 5mo ago

lucamark 5mo ago
Congrats! Most people won’t want to debug drivers, kernels, ACPI, adapters, and fan headers. But for those who do, the capability-per-pound is absurd.
lelanthran 5mo ago
> The compute is still real. The VRAM is still real. And the memory bandwidth is where it gets genuinely surprising. Because humans write exactly like this /s
- postalrat 5mo ago
  Where do you think llms learned to write that way?
  - alehlopeh 5mo ago
    Marketing content.
  - jlund-molfese 5mo ago
    You can also look at past posts by the same author (before LLM usage proliferated) if you’re curious. The project is still very cool, but it’s a little less enjoyable to read when everything sounds the same. It would be just as annoying for people to manually write in a corporate/marketing style, because humanity is what makes the small web interesting. https://blog.tymscar.com/posts/privategithubcicd/ https://blog.tymscar.com/posts/privategithubcicd/
    - iugtmkbdfil834 5mo ago
      This, setting aside the llm issue, it is dealing with hardware in ways that -- one would think - would be celebrated on HN of all places. But we focus on presentation.
    - tymscar 5mo ago
      I’m glad I’ve started this blog before the AI wave so I can prove people I’m just weird at writing. It grinds my gears how so many people just talk about my writing style instead of the content.
      - lelanthran 5mo ago
        > I’m glad I’ve started this blog before the AI wave so I can prove people I’m just weird at writing. Your previous blog posts didn't trigger any LLM detector (go on - check for yourself).
        tymscar 5mo ago
        Neither does this one. I replied in another thread. It comes out as 0% and the one from 2021 comes out as 8%. LLM detectors are all BS
        lelanthran 5mo ago
        GPTzero says 100% AI generated for specific paragraphs that I chose (such as `Multi-token prediction`). If you remove all the code listings, tables, etc and just paste the prose into these tools, it drops to 87% AI generated. None of the 3x older blogs of yours that I tried went above 5% AI generated. Maybe you're spending so much of time with the LLM that you are talking like it; in which case, take an old blog and a recent blog, give the prose from them both to you favourite LLM and ask them if the same author wrote both. I just did that on ChatGPT and on Gemini, and both found that it is extremely unlikely that the same author wrote both. Look, if all the SOTA LLMs agree that your recent blogs sounds generated, you can't blame the reader, can you?
        tymscar 5mo ago
        GPTzero is a joke. It thinks this is AI: “I bought a datacenter GPU that doesn’t even have a normal PCIe connector, stuck it in my gaming PC with an adapter, and now I have 32GB of VRAM across two GPUs running a 27 billion parameter model at 32 tokens per second.” There’s nothing AI about that. Not all SOTA LLMs agree, hell, none of them do. The same exact example I sent here gives me 0% in some, 10% in others, 100% in GPTzero.
  - lelanthran 5mo ago
    > Where do you think llms learned to write that way? Not from individual human content, that's for sure - maybe MLM marketing copy? Sleazy 4AM ads? I mean, every time this response comes up, I keep asking the person to point at something written prior to 2022 that gets 80%+ on the LLM detectors, and yet no one can find anything. Maybe you, postalrat, can find something written in this style that was published prior to 2022.
    - hattmall 5mo ago
      It's a function of the LLM "thought process"! It's not really modeled after human speech. It is in short segments but not long form, same reason you see the same rather odd nuances in LLM generated code. If they way you thought was to run a bunch of if statements, generate content, then feed that content back to get a "score" of what seems the most plausible, run the if statements again, and adjust / merge responses, then you would write similarly. The recognizable cadence of LLM generated content is pretty clearly the result of a lot of if statements being fused together.
    - tymscar 5mo ago
      I have written the blog post. I know empirically that I have used 0% AI while writing it. I also know LLM detectors are total BS and they don't really work. I have tried a couple on this exact blog post, and QuillBot, for example, gave me 0% AI detected on it. I have then used a blog post of mine from 2021. QuillBot gave me 8%... The King James version of the Bible came out at almost 100% AI generated a while ago. It was the HN front page. Stop thinking that if someone writes in a way that is fun or looks like what you would think an AI writes, then it is AI generated. Loads of the time it is, but sometimes it's not, and it really hurts those like me.
      - lelanthran 5mo ago
        > I have tried a couple on this exact blog post, and QuillBot, for example, gave me 0% AI detected on it. Don't use Quillbot; not sure why, but their model is reluctant to classify anything as AI generated. I ran into this when proof-reading a students Phd - ChatGPT, Gemini, CLaude (and others) all agreed it was AI generated, but Quillbot said it wasn't.
  - tgv 5mo ago
    Because their custom training data contains an emphasis on such verbiage. It doesn't come from the God-knows-how-many TB of web content the model is pre-trained on. There, such phrasing is only a drop in the sea. But the "yes, you're right" phrases, the em dash, etc., come from the later stage, for which content is created according to some (probably overprecise) guidelines.
    - rafram 5mo ago
      Right. The overuse of "genuinely" most of all. Seems like they put Claude through a few good rounds of training to always answer questions about its consciousness, thoughts, etc., with something about how it's "genuinely unsure," and as a result, the model learned to use "genuinely" as an intensifier in all sorts of inappropriate contexts.
      - iugtmkbdfil834 5mo ago
        Oi, I personally use adverbs everywhere. Genuinely, kids these days.
    - anigbrowl 5mo ago
      It's a very specific style of condescending journalism that US media has been nurturing and recycling for decades now. I was going to write this this whole comment as a parody of it, starting with some literary hook like 'Call it Ouroboros syndrome:' but I can't bring myself to add to the pile. I have not done the textual and statistical analysis to verify this, but I feel like it's something you could trace back to east coast journalism schools and publishers mediated via television, which long predates mass adoption of AI. Think how many news articles you've read with titles like 'Anatomy of a murder' os 'Inside the meeting that changed everything.' The hooky, slightly pompous tone is something you can find back as far as the 1960s or 1970s; browsing through old issues of Readers Digest and you'll find tons of it. When I say it's mediated through television, I'm talking about both the dramatic and heavily conclusory style of fictional prosecutors and narrators, and the extremely shallow style of TV news reports (often transcribed to the web) which are only one or two sentences per paragraph. And this is before we consider the stylistic impact of ad copywriting on communication in general. And there's something else. The one sentence paragraph interjection, designed to refocus your attention in a surprising new direction after two paragraphs of stuff you already know. 'I never thought I'd end upere,' said Sally Nocontext, hooking you in for another paragraph or two where you try to figure out who this woman is, where she ended up, and what it has to do with the article you are already halfway through reading. After all, I've come this far, the reader through. I might as well see it through to the end. And that's just what publishers wanted. One sentence can also validate a truism that the reader already suspects, flattering their beliefs in their own analytical powers.... ...well you get the idea. When I'm using LLMs for any sort of extended session, I find myself reaching for the same few prompts to break it of such clicheed expression; I'm especially averse to the habit of adding zippy-sounding nicknames to complex or potentially dull concepts. I don't have a favorite starting prompt, but I generally find that asking for 'a concise, academic tone' does wonders to de-fluff its output. Remember, it defaults toward being as widely accessible as possible, and much journalism is aimed at consumers with only a high school education and maybe middle-school reading comprehension, math ability, and appetite for depth over sensation.
  - krackers 5mo ago
    This particular case seems to be an LLM trying to blindly apply the saccharine therapeutic pattern "your frustration is real" to a context where it doesn't make sense. No one is debating or questioning the fact that the card has 16 gigs of vram. The point of "X is real" in a therapeutic context is to make the person feel seen and acknowledged, that his struggles are real to him and really do weigh on his mind, even if it is technically "all in his head".
  - LearnYouALisp 5mo ago
    It goes both ways, too -- people see all this on Medium, elsewhere, even in the comments on threads in YC
- bossyTeacher 5mo ago
  X is Y. Z is Y. And Alpha is genuinely Beta. Classic LLM writing style.
- driverdan 5mo ago
  There's interesting stuff in this writeup but it sure seems like most of it was written by an LLM.
- bitwize 5mo ago
  You know what the sad bit is? Humans do write exactly like that. That's not even particularly egregious StalkedIn marketroid speak.
matja 5mo ago
The AMD MI250X GPUs are also interesting - 128GB of HBM2E at 3TB/s, sometimes you see them second-hand for under $1k, the catch obviously is that it needs an OAM socket. Never seen an easy way to hook them up to a regular mainboard.
- Teknomadix 5mo ago
  These are interesting, and offer beefy through put. No point in adapting to a PCI lane thought, stuck behind the slot-bus bottleneck.
- Gracana 5mo ago
  An additional complication is that MI250Xes are two GPUs in one package, so you need to connect the first and last x16 SERDES groups to the host, otherwise you'll only see one GPU (or it won't work at all, idk). Also, the cheap HPE pulls on eBay need some proprietary HPE magic to work, and I have yet to see anyone figure that out.
- plagiarist 5mo ago
  Ahh luckily this OAM socket will prevent me from spending money.
  - selectively 5mo ago
    [dead]
- sonzohan 5mo ago
  This person has built a converter for the OAM socket, but it is only confirmed working with NVIDIA cards at the moment (https://www.reddit.com/r/NVIDIA_SXM2PCIE/comments/1d076cn/oam_to_pcie/ https://www.reddit.com/r/NVIDIA_SXM2PCIE/comments/1d076cn/oa...) It fits an MI250X, and the system sees it, but the drivers don't work. They tested an HPE MI250X. There's a rumor on the thread that there are two kinds of MI250X: Ones from HPEs and everyone else's. The HPEs require a special firmware, the normal ones do not. However, the majority of the MI250Xs on the secondhand market are HPE so caveat emptor.
knollimar 5mo ago
A little bit of local copium but neat read. Isn't a rasbpi with 16gb of RAM $300 now?
- matja 5mo ago
  The latest Raspberry Pi 5 has one 32-bit channel (2x 16-bit subchannels) of LPDDR4X-4267 SDRAM giving 17.1GB/s of bandwidth, 52x less than this GPU. Never mind lacking the CUDA and Tensor cores, so the FP16 performance is 102x less (307 GFLOPS vs 31.4 TFLOPS). So for £200, there's absolutely no comparison for this specific use-case.
  - knollimar 5mo ago
    Yeah thats what I'm saying. How is it so cheap????
    - feisuzhu 5mo ago
      V100 GPUs are e-waste.
- thejj100100 5mo ago
  I don't understand what point you're trying to make here? Are you talking about the price of RAM?
jmyeet 5mo ago
Some context: - In 2017, the v100 was a ~$10,000 GPU. I believe there was a PCI-e version but this is probably so cheap because SXM2 is going to be harder to use; - A 5090 has 1800GB/s of internal memory bandwidth (compared to 900GB/s in the 9 year old GPU). Of course a 5090 is substantially more expensive; - A 5090 has ~21k CUDA cores vs ~5k; - The current $10k NVidia GPU is the RTX 6000 Pro w/ 96GB of VRAM. It has slightly more CUDA cores but it otherwise pretty much just a 5090. This is unsurprising. NVidia uses VRAM for market segmentation. Consider this: in 5-10 years, the trillions spent on AI data centers will likewise be sold for scrap most likely. That's how short the runway is for OpenAI and Anthropic to recover that investment. Anyway, I'm kind of impressed the author managed to get this all to work. I don't think it even would've occurred to me that someone had made an SXM2 adapter, particularly because it's not even used anymore. Like props to whoever did that.
- b112 5mo ago
  I bet 3 years, but otherwise agree.
- echelon 5mo ago
  > Consider this: in 5-10 years, the trillions spent on AI data centers will likewise be sold for scrap most likely. That's how short the runway is for OpenAI and Anthropic to recover that investment. Even more interesting: it'll devalue all of SaaS and the entire US tech sector. We might have just shot our most valuable non-AI tech products in the foot.
  - wholinator2 5mo ago
    How so? I understand that flooding the market with physical goods will reduce prices and thus profits. But how would that also reduce the nonphysical SAAS stuff?
    - mschuster91 5mo ago
      > But how would that also reduce the nonphysical SAAS stuff? The resulting economic crash will affect everyone, we're (IMHO) looking towards a dotcom-bust level wipeout. And many SaaS and other companies run asset-lean (i.e. they have no server hardware because that's all cloud, no real estate because it's all either wework or conventionally rented), margin-lean (the VC business model requires that, as the basic recipe is to achieve market domination by burning cash) and cash-lean (often enough, it's less than a quarter of expenses on the bank accounts). All that "lean-ness" looks great on an investor's quarterly release sheet: no massive amounts of wealth tied up in assets and no cash sitting around on bank accounts that could be released towards investors as dividends or, if it comes from third parties, costs the company interest... but it prevents resiliency against crises.
mondainx 5mo ago
Great write-up, I've often considered these DC cards for a project and now you've convinced me to pick one up; you describe the price of the unit against what one spends on tokens and that does it for me.
- tymscar 5mo ago
  Thats why I did it. I think it’s important to put things like that into perspective
casey2 5mo ago
Some resell group is going to have to make this easier. The shear amount of these cards otherwise heading towards the landfill is staggering. That is if Big Tech don't destroy them to prevent model weights from leaking.
- eric__cartman 5mo ago
  How would destroying the GPUs prevent the model weights from leaking? By the time you get your hands on them the memory is powered off for a long enough time that a cold-boot style attack is impossible.
  - sethops1 5mo ago
    Would you bet your trillion dollar company on that? Or would you smash up the garbage [to you] memory chips to be sure.
    - marcosdumay 5mo ago
      It's volatile memory, not flash.
    - tymscar 5mo ago
      I would bet my trillion-dollar company on it because I understand how RAM works.
- Alifatisk 5mo ago
  > The shear amount of these cards otherwise heading towards the landfill is staggering. The thought of throwing away working cards sounds so bizarre to me. I can't believe companies would dispose them into the landfill like that, it is at least worth giving away for refuse.
  - wookmaster 5mo ago
    There’s a long history of corporations doing evil things to ensure their business model succeeds
Teknomadix 5mo ago
Tesla V100 SXM2 16GB is NOT DGX class as the author writes. It's HGX class. The V100 comes in two classes, SXM2 and SXM4, the latter coming with a Max of 80gb on board memory. Typically these are installed 8×A100 80GB SXM4 on an HGX riser, and what that gives you is NVSwitch fabric and 640GB of pooled HBM2e (on package stacked memory /w ~2 TB/s of memory bandwidth). 2u standard rack footprint too.
- legitronics 5mo ago
  I have no idea what you are trying to say. V100 came as sxm2 and sxm3. And it was 16 and 32gb. HGX is DGX with extra toppings.
  - _zoltan_ 5mo ago
    No. DGX is when it's Nvidia's design, HGX when it's a 3rd party design.
- _zoltan_ 5mo ago
  What on earth are you talking about? Your comment makes no sense. The V100 and A100 are different generations altogether. The V100 does not have 2TB/s.
omarqureshi 5mo ago
Could probably avoid the crazy fan with a waterblock - I've seen a whole kit, v100 + PCIE adapter + block for £235. Yes, you'll have to pay for pump, radiators and radiator fans, but that should really quieten it down
- pogue 5mo ago
  Someone's already made such a kit as you describe to fit in a consumer PC case and work properly?
  - deleted 5mo ago
    [deleted]
  - omarqureshi 5mo ago
    https://www.ebay.co.uk/itm/406939340557?var=677143153030&mkevt=1&mkcid=1&mkrid=710-53481-19255-0&campid=5338749390&toolid=20006&_ul=GB&customid=CjwKCAjwuO_QBhAWEiwAIkVhU6ne-qhrP48l-uTWETS88KvgKcphn7XUTPUriwgtA8YHYhG2kEiffhoCAm0QAvD_BwE|0AAAAADihhqWHf8GvoHUZ0mwHZX98FeGvV|CkAKCAjwlerQBhAVEjAAO1hnVkWOFBn0RwIjpOQilhR03KVRGI_V6TeI0pxjEH3-ddHuJ0rf_RA_ogU38eIaAsf0&gclid=CjwKCAjwuO_QBhAWEiwAIkVhU6ne-qhrP48l-uTWETS88KvgKcphn7XUTPUriwgtA8YHYhG2kEiffhoCAm0QAvD_BwE&gbraid=0AAAAADihhqWHf8GvoHUZ0mwHZX98FeGvV&wbraid=CkAKCAjwlerQBhAVEjAAO1hnVkWOFBn0RwIjpOQilhR03KVRGI_V6TeI0pxjEH3-ddHuJ0rf_RA_ogU38eIaAsf0&loc_interest_ms=&loc_physical_ms=9045504&adtype=pla&gad_source=1&gad_campaignid=17335954774&gbraid=0AAAAADihhqWHf8GvoHUZ0mwHZX98FeGvV https://www.ebay.co.uk/itm/406939340557?var=677143153030&mke...
  - omarqureshi 5mo ago
    32 GB kit as well https://www.ebay.co.uk/itm/406939344915?_trkparms=amclksrc%3DITM%26aid%3D1110006%26algo%3DHOMESPLICE.SIM%26ao%3D1%26asc%3D303466%26meid%3Dfb09c9c6d3e04fd8b809446bd02877df%26pid%3D101224%26rk%3D2%26rkt%3D5%26sd%3D406939340557%26itm%3D406939344915%26pmt%3D0%26noa%3D1%26pg%3D2332490%26algv%3DSimOrganicCassiniWithToraRecalls%26brand%3DNVIDIA&_trksid=p2332490.c101224.m-1&var=677143082490 https://www.ebay.co.uk/itm/406939344915?_trkparms=amclksrc%3...
deleted 5mo ago
[deleted]
hypfer 5mo ago
[dead]
recursivegirth 5mo ago
> The compute is still real. The VRAM is still real. And the memory bandwidth is where it gets genuinely surprising. Had to stop there. Annoying. I can't stand AI use for writing. It makes any otherwise great article feel so disingenuous.
- m0rde 5mo ago
  What a difficult world you must live in these days
  - peddling-brink 5mo ago
    While I don’t disagree with their sentiment, I’m far more annoyed with it than the AI writing.
    - m0rde 5mo ago
      Yeah. I get that many HN comments are just complaints (heck mine was too and just as negative and shaming). But how bad of a day must you be having to try to shame someone about how they choose to write up an experience they thought was neat. Whatever, free speech and all that. Hope OC's day gets better.
      - gsquaredxc 5mo ago
        It doesn’t read like shaming to me. It’s, in the grand scheme of HN comments, definitely on the more constructive side of the criticism. Maybe it could have been reworded, but I think the author of the post could very easily find it actionable in the future. I too had to stop reading the article at that point, so I think if the author wants more people to read, my advice for them is to just write like themselves. We’ve entered the start of a new Instagram filter age where many people feel they need to have LLMs reword their writing presumably for the same reasons as the original filter age. I share OC’s sentiment of pushing against the recent trend of implicitly shaming people for their individualistic writing styles.
    - qingcharles 5mo ago
mickeyp 5mo ago
Impressive work. But the problem is not the 30 tok/s which is fine for agentic coding and chat. It's prefill; slow prefill kills agentic workloads dead. If you have 100,000 tokens at ~150tok/s per the OP, you're looking at: You have: 100000 / (150/s) You want: hms 11 min + 6.6666667 sec Which is quite a wait indeed.
- Aurornis 5mo ago
  Most people won’t be dumping 100K tokens into it at once, but I agree that all of the prefill time that adds up during a session becomes a lot to account for. This is also a problem for all of the Mac local LLMs. Macs are a great way to get a lot of high bandwidth memory, but their compute is very far behind current gen dedicated GPUs. Some of the expensive Mac Studio setups allow you to run very large models with usable tokens/s, but you can be waiting a long time for it to get to the point of generating those tokens.
  - Tepix 5mo ago
    When you're using OpenCode it's easy to reach 100,000 tokens after a while.
  - pyrolistical 5mo ago
    The prefix cache is working properly 100k doesn’t prefill more than once
- HarHarVeryFunny 5mo ago
  I wonder if this could be usefully mitigated with a combination of prompt (prefix) caching and an agent that let you control what the prompt prefix consisted of. The goal would be to incur that slow prefill once to build the prompt cache, then have subsequent prompts consist of mostly this fixed prefix plus specific instructions. For a language like C++ where modules are split into definition (.h) and implementation (.cpp) parts, one choice of prefix would be all the header files for the project (which aren't likely to change much). More generally the idea would be to have an agent that had cached-prefix reuse as it's primary context management goal. Another possibility, to support caching of files that have since changed, would be for the agent to build the context as a fixed prefix reflecting some or all of the codebase in its start-of-session state, then append any changes to that, with appropriate prompting to only use the latest definition of a function. e.g. Say file A initially contains functions X, Y and Z, then the prompt prefix is built to include X Y Z. If the user then modifies Y -> Y', then just add that to the context, so that the cached prefix is unchanged, giving X Y Z Y'.
whoamii 5mo ago
The real question: did your local LLM write this post?
- 20wenty 5mo ago
  There are many tells aren't there? There was clearly hard human work and experimentation here, but it's a shame the OP let AI do chunks of the writing. Once you see it, it's much harder to take the post seriously.
  - iugtmkbdfil834 5mo ago
    I disagree. Not everyone has a good writing style. In those instances I think it is fair to default to llm recommendation. We may be allergic to it, but we saw one formulaic response too many ( though admittedly it does raise a question of whether HN was the intended audience for it ). In any event, not all of us have a unique writing style worth preserving just like not all of us can write clear and clean code. Just saying.
    - unshavedyak 5mo ago
      I really wish it was more common to use AI for augmenting than authoring. Eg i find coding with LLMs neat when you primarily "talk" to it through code, by filling out structs, funcs, fields, etc - where it would use your changes as the template and then to work to effectively autocomplete the gaps. The more you iteratively write the less it fills in, but also the less it deviates from your intent, design, etc. I feel like writing could use a similar harness, where it attempts to minimally reword the authors sentences, perhaps just tweaking grammar, spelling, etc. In the coding example i think the human code would be near unchangeable, the LLM would pivot around it - but in the writing example i think the human writing would have to be more mutable. I imagine it would be a configurable setting. I've not really seen a system which focuses on this human<->LLM look, but it feels interesting to me.
      - iugtmkbdfil834 5mo ago
        In a sense, there is a clear market for it ( people want 'authentic' experience ). I can kinda understand it. I want pure linux experience without systemd, but I recognize that in the current ecosystem, it comes at a cost. So the language harness makes sense to me, but corps are already cracking down on token use ( and such a harness would likely only add to the cost ). The other question is whether the people, who could benefit it would even recognize it as a problem though.
bob1029 5mo ago
> And yes, if you want the absolute best, Opus 4.8 exists. It also costs more per 20 minutes of heavy use than I paid for this entire GPU and adapter setup combined. But the gap is shockingly small. I don't think this is a fair characterization of the situation. I use frontier models via API pre-paid tokens every single day, and I can barely rack up $100 per month. The fact that we figured out how to burn double this in 20 minutes is impressive, but I don't think it reflects the reality that many are experiencing right now. There are some exceptionally gluttonous approaches to harnessing LLMs that I think are serving as convenient straw men in these discussions. Paying for the API will almost always be more economical than self-hosting equivalent infrastructure. I am not against self-hosting, but the article suggests a primarily economic motivation for this effort. If you are consuming fewer than 10^9 tokens per month, I really don't think it's worth your time to try and compete with the hyperscalars. Most of the money is to be found in the integration of this technology with existing businesses.
- vidarh 5mo ago
  I use hosted providers myself, but I can churn through $100 worth of tokens in half a day even with cheap models like Deepseek easily. If someone's use is as light as yours, then sure - grab a subscription and you'll save far more. For higher use it will come down to how cheap your electricity is whether it is worth offloading at least some of it (for me it's not, FWIW)
  - iJohnDoe 5mo ago
    Could you share a bit about what you’re working on or what type of projects require that much usage? Is it hobby, production, revenue generating?
    - vidarh 5mo ago
      A mix. I have hobby projects that churn through that much when I don't need the tokens for others things. I also have projects for clients that easily consumes those levels. As well as a stealth-ish potential startup. Currently I'm at 4 different subscriptions + more than I'd like in spend via OpenRouter... What multiplies it very quickly is when you start feeding them with test suites and "Ralph loops" that run until the test suites pass, or complex chains with lots of sub-agents being triggered. If you're sitting there watching everything, it will be hard to burn all that much even if you're running multiple things in paralle.
ewy1 5mo ago
despite gaming being used in the title, it is not mentioned in the article, but i'm curious how this performs. i've ran some multi vendor frankenstein setups before and sometimes it even works, so i'm curious to hear your experience with it.
pogue 5mo ago
But could you game with the GPU? Or is that purely a drivers issue?
- mcraiha 5mo ago
  I assume you can game with it if you use modded drivers. At least with CMP 50HX that is possible https://videocardz.com/newz/nvidia-cmp-50hx-turned-into-gaming-gpu-shows-lackluster-performance-due-to-limited-pcie-lanes https://videocardz.com/newz/nvidia-cmp-50hx-turned-into-gami...
abejfehr 5mo ago
Based on the title I was really hoping to see how this was used for gaming, but they just ran an LLM on it
- axpy906 5mo ago
  Same. With no new NVIDIA gaming GPUs this year, seems like an interesting problem to solve.
- mschuster91 5mo ago
  I don't think that is even possible, every piece of silicon on that chip that is required to do gaming is ripped out in favor of more compute cores.
- darkwater 5mo ago
  They said in the beginning that it doesn't even have a video out, so you cannot do gaming.
  - yjftsjthsd-h 5mo ago
    I thought you could run games by rendering on one GPU and outputting on another? Usually comes up with dual iGPU/dGPU setups, but could work here
  - toast0 5mo ago
    I've seen things where you have multiple video cards and can use one gpu to render to a framebuffer which is transferred to the other video card to output. I'm sure it adds latency, and it's probably unsupported... But no output doesn't mean can't do gaming... It just means gaming will be iffy. There's some virtualized desktop server stuff too. Run a bunch of desktop sessions on a beefy computer and send a video stream to desktop players. With the right codec settings, the latency is probably ok for many games.
  - hakfoo 5mo ago
    I'm actually surprised there hasn't been a dedicated effort to support display offload to, say, the CPU's iGPU. I'm sure manufacturers would love saving a dollar per card, and OEMs would appreciate eliminating the support calls from "I just bought a new $2500 gaming PC and no video" because they plugged the monitor into the iGPU instead of dGPU.
wg0 5mo ago
Wait a few years, everyone will be able to put one at half the price.
KnuthIsGod 5mo ago
AI written posts will kill HN.
- tymscar 5mo ago
  AI didnt edit a single word of this post.
axpy906 5mo ago
Wow. V100. That brings back memories. Way to go.
gtirloni 5mo ago
> The compute is still real. The VRAM is still real. And the memory bandwidth is where it gets genuinely surprising. sigh
- krackers 5mo ago
  Usually that cloying pattern is reserved for "emotional" contexts to validate the user ("your struggles are real [despite others thinking it's in your head]"). Here it doesn't even make sense, of course the VRAM is real. Is it going to tell me that my keyboard is real next? I wonder if this was generated with the local model, this seems to be a case where it memorized the style but not the meaning and intent.
viseyth 5mo ago
Volta (and Pascal, which I'm using) should still be supported with driver 580 as long as you don't use the open modules, and you can use up to cuda 12.9 and cudnn 9.10.2. No need to limit yourself to an old kernel.
- markus92 5mo ago
  It is. We still run quite a few of them in prod and with 580 drivers they run just fine. Very useful GPUs still.
segmondy 5mo ago
The most interesting and perhaps useful for most would be how they control the fan. If you are thinking of doing this, you really want to get those fans under control, they are loud. For anyone thinking of these, v100s idle super high! 25-35watt with nothing loaded and easily 50w when a model is loaded.
00dazzle 5mo ago
That's the same price per VRAM GB as an arc pro B70
- tymscar 5mo ago
  But with miles better support, thats why I went this route. Cuda is hard to beat
jonhohle 5mo ago
I was just looking into this and was worried about the fan setup. Interesting that he was able to solve it with good results. In case anyone is interested, I’m using PCIE passthrough on a FreeBSD host to a Linux guest with an older Pascal card. It’s worked great and I’ve been thinking about putting a nicer card in there. The SXM route seems great, but I’ve been burned (almost literally because of the heat) by DC components before.
sonzohan 5mo ago
I also recently decided to buy a datacenter GPU and slap it into a system. Some notes from my experience that the author doesn't mention in their article: Decommissioned NVIDIA V100s and AMD MI50s are fairly cheap, $200 for 16gb and $400-500 for 32gb, for local experimentation. They are also very old. There's an enthusiast community keeping these two cards alive and working with current platforms and models. Nitpick, but the V100 doesn't support bfloat16. The performance hit is not a big deal if you're fiddling with local models, but the card is on it's way out in terms of hardware features. The MI50 does support bf16, but not the current edition of AMD ROCm. Vulkan support is good and the MI50 works with most major platforms (llama.cpp, vllm, etc.), but it's not without some pain points like manual recompilation. Fortunately the open source community has already paid most of your way. The cooling requirements for these cards cannot be understated. A consumer grade GPU may throttle if in a small case without additional fans, but if given the same treatment a datacenter GPU will overheat itself idling. You will need to buy, at least, a bunch of decent 120mm fans to prevent this or invest in some water cooling. I ultimately went with an AMD MI100 32GB ($950). I'm an AMD fan, current ROCm editions support it, and it was low-fuss to get things working. I'm debating getting a second so I can try out bigger models like qwen3-coder-next.
- Silagi 5mo ago
  Did you consider the R9700 or B70 when you went for the MI100? If so, what made you choose the MI100? I've been playing with picking up a card in this class but haven't been able to justify it when running the Qwen3.6 MOE model on a 6800xt is tolerable for the type of projects I've been willing to point local AI at.
  - sonzohan 5mo ago
    I looked at those, the Arc 1100, the w6800, MI50, MI60, v100, v620, and basically anything with 32gb of RAM: 1. I wanted an AMD card. 2. I have an RTX 3090 that's been fun to play with, but I want to get back to using it for gaming. 3. I was looking for between 30-60 tokens/second in terms of performance on the beefier models I want to run. Looking at stock Qwen3 32B the benchmarks reported about 41 tokens/second for MI100. w6800 was 18, MI50 & MI60 could do 60s but had a lot of compromises/special things to achieve that. 4. I used FitMyLLM for some spec-based comparisons (https://www.fitmyllm.com/ https://www.fitmyllm.com/). The MI100 is roughly double the performance on Qwen 3.5 35B A3B Q5_K_M to the R9700 (462 token/s prefill vs 239 tokens/s, 217 tokens/s vs 118 token/s for inference) 5. I was willing to throw up to $1k at a GPU; I really wanted to throw closer to $650. To be honest, if money was no objection I would've sprung for a MI210. I also considered the MI250 as they showed up for $1250-1400 with a whopping 128GB, but the PCIE converters for that form factor don't have working AMD drivers yet.
melonpan7 5mo ago
Great value for money if you have the time for tinkering and getting the compatibility to work.
- DANmode 5mo ago
  and it only gets easier and more defined from here.
- tymscar 5mo ago
  If you use NixOS, you don’t need to do any of that anymore. A single command in the CLI, and you have everything I do.
mettamage 5mo ago
> The way it works is that a vision encoder (similar to what ChatGPT and Claude use) takes image pixels and translates them into the LLM’s token embedding space. The model does not “see” the image the way a human does. Instead, the vision encoder compresses the image into a sequence of vectors that live in the same mathematical space as text tokens. The LLM then processes those vectors as if they were just another sequence of tokens. Could you also do this for music and specifically sound synthesis? It would be awesome to vibe synthesize sounds and then see the VSTi parameters surrounding it.
- Ey7NFZ3P0nzAe 5mo ago
  I don't think so. Cramming new senses into the latent space of the model is one thing, but having a model output tokens that can be detokenized into sound is completely different and requires a very different type of data.
  - LearnYouALisp 5mo ago
    What do you mean? Why not? We can already FFT sound into "words", so why not have some kind of dictionary to an arbitrary level of precision/fidelity?
    - Ey7NFZ3P0nzAe 5mo ago
      Sound frequency is continuous : "between sound of A and O" means something, but "between the letter A and F" means nothing.
jeffrallen 5mo ago
Super interesting. I use data center GPUs at work, but I didn't know anything about this stuff. I also use Qwen 3.7 27b at work and I agree with the author: it is perfectly capable of the jobs I give it.
peibye 5mo ago
All that work just to write an ai blog post. This is a cool topic but I just can’t deal with the aiisms.
drumhead 5mo ago
The first dual core processor I built a machine with was an Opteron. It was a nice piece of hardware.
- LearnYouALisp 5mo ago
  xw9300 fan
suralind 5mo ago
This is great! I've been trying to get into local models for a while as I share the sentiment that local models will eventually be so good that there won't be a need to use frontier models for most coding tasks (perhaps that's already true today?). I have zero experience building computers - where would I even start? I mean, aside from the things already well documented and mentioned in the blog post.
- tymscar 5mo ago
  Building computers is very easy. I would suggest watching a YouTube video to get the general gist, and then once you buy the parts, just Google for whatever doesn’t go well. I built my first Pentium 4 one when I was like six, so I’m sure someone much older that’s into tech can do it without an issue. There are also tons of Discord communities that are willing to help you live if you encounter any issues.
binyu 5mo ago
The V100 and the 4090 are based on vastly different architectures, the former uses the older Volta while the latter uses Ada. Last I checked you cannot meaningfully combine them. The 3090 is better than the V100, just get two 3090 and a NVLink.
- tymscar 5mo ago
  Well I did in fact meaningfully combined them without an issue, that was the whole point of the blogpost.
  - binyu 5mo ago
    Yes but it creates a bottleneck that negates the benefit of using multiple cards that way. Look into it. Cheers
    - tymscar 5mo ago
      Well it doesn’t matter because the bottleneck here is actually quite small for me. The issue is vram. If anything the bottleneck is my 4080.
      - binyu 5mo ago
        Gotcha, I am not saying your setup is inherently wrong or useless. I am glad it works for your use cases. Godspeed
        tymscar 5mo ago
        I think its a very fair thing you have flagged!
- cthalupa 5mo ago
  You can split tensors across an AMD GPU and Nvidia GPU - different architectures are not an issue. People run LLMs across some pretty crazy setups.
QuantumNoodle 5mo ago
This is very cool. Can anyone point me to any resources for setups that can ron decent inferencing at home?
mg794613 5mo ago
Very interesting, but slight note on the fact that writer sort of forgets to calculate in the price of the 4080 it works together with. He spend 200 to upgrade his existing setup, great nonetheless. But not "32gb gram for only 200 bucks"
- tymscar 5mo ago
  I did mention that in my case I had the 4080, but you do not need the 4080 whatsoever. You can run the same model with less context on a single £200 V100 or with the same amount of context on two of these.
rbanffy 5mo ago
This lack of driver support for hardware that’s still functional and available in reasonable quantities is annoying. I was very close to buying a retired POWER7+ server with an ungodly amount of memory, but decided being unable to run a modern Linux kernel would be more work than I wanted to have. Modern kernels need POWER8 and above.
rbanffy 5mo ago
This lack of driver support for hardware that’s still functional and available in reasonable quantities is annoying. I was very close to buying a retired POWER7+ server with an ungodly amount of memory, but decided being unable to run a modern Linux kernel would be more work than I wanted to have. Modern kernels need POWER8 and above. OTOH, if these chips were fully supported, they wouldn’t hit the second hand market at the prices they do.
neutrinobro 5mo ago
Very nice to see this older hardware getting repurposed. I have been running 2x Tesla V100s in a dual-core supermicro X10DRU-i server. With qwen3.6-27B-mtp I get about 35-40tok/s for inference for moderate context sizes (<128k), and have run long running agent tasks on it which consume 100s of millions of tokens (>$100s if I had to pay claude API costs). However, the main purpose that I have to for these cards is for scientific compute, the FP64 performance (7+ TFLOPS!) is fantastic given their age, and not something you can get on even the latest consumer grade cards since Nvidia nerfed their performance after Kepler. The server lives in the basement though...it is freaking loud!
Ey7NFZ3P0nzAe 5mo ago
While we're at it, has anyone any tips on turning a consumer RTX 3090 into a blower type format for putting inside a server rack? Any kind of workaround like this.
j4k0bfr 5mo ago
Thank you for writing this! And sharing the NixOS config. I've been looking for an excuse to get into Nix for ages, but this might finally do it. P.S. Man the AI writing complaints on this thread are quite upsetting. OP is handling it a lot better than I would, lol.
AI2070 5mo ago
now design a shroud and blower

332 points by birdculture 5mo ago

Congrats! Most people won’t want to debug drivers, kernels, ACPI, adapters, and fan headers. But for those who do, the capability-per-pound is absurd.

lelanthran 5mo ago

> The compute is still real. The VRAM is still real. And the memory bandwidth is where it gets genuinely surprising. Because humans write exactly like this /s

postalrat 5mo ago
Where do you think llms learned to write that way?
- alehlopeh 5mo ago
  Marketing content.
- jlund-molfese 5mo ago
  You can also look at past posts by the same author (before LLM usage proliferated) if you’re curious. The project is still very cool, but it’s a little less enjoyable to read when everything sounds the same. It would be just as annoying for people to manually write in a corporate/marketing style, because humanity is what makes the small web interesting. https://blog.tymscar.com/posts/privategithubcicd/ https://blog.tymscar.com/posts/privategithubcicd/
  - iugtmkbdfil834 5mo ago
    This, setting aside the llm issue, it is dealing with hardware in ways that -- one would think - would be celebrated on HN of all places. But we focus on presentation.
  - tymscar 5mo ago
    I’m glad I’ve started this blog before the AI wave so I can prove people I’m just weird at writing. It grinds my gears how so many people just talk about my writing style instead of the content.
    - lelanthran 5mo ago
      > I’m glad I’ve started this blog before the AI wave so I can prove people I’m just weird at writing. Your previous blog posts didn't trigger any LLM detector (go on - check for yourself).
      - tymscar 5mo ago
        Neither does this one. I replied in another thread. It comes out as 0% and the one from 2021 comes out as 8%. LLM detectors are all BS
        lelanthran 5mo ago
        GPTzero says 100% AI generated for specific paragraphs that I chose (such as `Multi-token prediction`). If you remove all the code listings, tables, etc and just paste the prose into these tools, it drops to 87% AI generated. None of the 3x older blogs of yours that I tried went above 5% AI generated. Maybe you're spending so much of time with the LLM that you are talking like it; in which case, take an old blog and a recent blog, give the prose from them both to you favourite LLM and ask them if the same author wrote both. I just did that on ChatGPT and on Gemini, and both found that it is extremely unlikely that the same author wrote both. Look, if all the SOTA LLMs agree that your recent blogs sounds generated, you can't blame the reader, can you?
        tymscar 5mo ago
        GPTzero is a joke. It thinks this is AI: “I bought a datacenter GPU that doesn’t even have a normal PCIe connector, stuck it in my gaming PC with an adapter, and now I have 32GB of VRAM across two GPUs running a 27 billion parameter model at 32 tokens per second.” There’s nothing AI about that. Not all SOTA LLMs agree, hell, none of them do. The same exact example I sent here gives me 0% in some, 10% in others, 100% in GPTzero.
- lelanthran 5mo ago
  > Where do you think llms learned to write that way? Not from individual human content, that's for sure - maybe MLM marketing copy? Sleazy 4AM ads? I mean, every time this response comes up, I keep asking the person to point at something written prior to 2022 that gets 80%+ on the LLM detectors, and yet no one can find anything. Maybe you, postalrat, can find something written in this style that was published prior to 2022.
  - hattmall 5mo ago
    It's a function of the LLM "thought process"! It's not really modeled after human speech. It is in short segments but not long form, same reason you see the same rather odd nuances in LLM generated code. If they way you thought was to run a bunch of if statements, generate content, then feed that content back to get a "score" of what seems the most plausible, run the if statements again, and adjust / merge responses, then you would write similarly. The recognizable cadence of LLM generated content is pretty clearly the result of a lot of if statements being fused together.
  - tymscar 5mo ago
    I have written the blog post. I know empirically that I have used 0% AI while writing it. I also know LLM detectors are total BS and they don't really work. I have tried a couple on this exact blog post, and QuillBot, for example, gave me 0% AI detected on it. I have then used a blog post of mine from 2021. QuillBot gave me 8%... The King James version of the Bible came out at almost 100% AI generated a while ago. It was the HN front page. Stop thinking that if someone writes in a way that is fun or looks like what you would think an AI writes, then it is AI generated. Loads of the time it is, but sometimes it's not, and it really hurts those like me.
    - lelanthran 5mo ago
      > I have tried a couple on this exact blog post, and QuillBot, for example, gave me 0% AI detected on it. Don't use Quillbot; not sure why, but their model is reluctant to classify anything as AI generated. I ran into this when proof-reading a students Phd - ChatGPT, Gemini, CLaude (and others) all agreed it was AI generated, but Quillbot said it wasn't.
- tgv 5mo ago
  Because their custom training data contains an emphasis on such verbiage. It doesn't come from the God-knows-how-many TB of web content the model is pre-trained on. There, such phrasing is only a drop in the sea. But the "yes, you're right" phrases, the em dash, etc., come from the later stage, for which content is created according to some (probably overprecise) guidelines.
  - rafram 5mo ago
    Right. The overuse of "genuinely" most of all. Seems like they put Claude through a few good rounds of training to always answer questions about its consciousness, thoughts, etc., with something about how it's "genuinely unsure," and as a result, the model learned to use "genuinely" as an intensifier in all sorts of inappropriate contexts.
    - iugtmkbdfil834 5mo ago
      Oi, I personally use adverbs everywhere. Genuinely, kids these days.
  - anigbrowl 5mo ago
    It's a very specific style of condescending journalism that US media has been nurturing and recycling for decades now. I was going to write this this whole comment as a parody of it, starting with some literary hook like 'Call it Ouroboros syndrome:' but I can't bring myself to add to the pile. I have not done the textual and statistical analysis to verify this, but I feel like it's something you could trace back to east coast journalism schools and publishers mediated via television, which long predates mass adoption of AI. Think how many news articles you've read with titles like 'Anatomy of a murder' os 'Inside the meeting that changed everything.' The hooky, slightly pompous tone is something you can find back as far as the 1960s or 1970s; browsing through old issues of Readers Digest and you'll find tons of it. When I say it's mediated through television, I'm talking about both the dramatic and heavily conclusory style of fictional prosecutors and narrators, and the extremely shallow style of TV news reports (often transcribed to the web) which are only one or two sentences per paragraph. And this is before we consider the stylistic impact of ad copywriting on communication in general. And there's something else. The one sentence paragraph interjection, designed to refocus your attention in a surprising new direction after two paragraphs of stuff you already know. 'I never thought I'd end upere,' said Sally Nocontext, hooking you in for another paragraph or two where you try to figure out who this woman is, where she ended up, and what it has to do with the article you are already halfway through reading. After all, I've come this far, the reader through. I might as well see it through to the end. And that's just what publishers wanted. One sentence can also validate a truism that the reader already suspects, flattering their beliefs in their own analytical powers.... ...well you get the idea. When I'm using LLMs for any sort of extended session, I find myself reaching for the same few prompts to break it of such clicheed expression; I'm especially averse to the habit of adding zippy-sounding nicknames to complex or potentially dull concepts. I don't have a favorite starting prompt, but I generally find that asking for 'a concise, academic tone' does wonders to de-fluff its output. Remember, it defaults toward being as widely accessible as possible, and much journalism is aimed at consumers with only a high school education and maybe middle-school reading comprehension, math ability, and appetite for depth over sensation.
- krackers 5mo ago
  This particular case seems to be an LLM trying to blindly apply the saccharine therapeutic pattern "your frustration is real" to a context where it doesn't make sense. No one is debating or questioning the fact that the card has 16 gigs of vram. The point of "X is real" in a therapeutic context is to make the person feel seen and acknowledged, that his struggles are real to him and really do weigh on his mind, even if it is technically "all in his head".
- LearnYouALisp 5mo ago
  It goes both ways, too -- people see all this on Medium, elsewhere, even in the comments on threads in YC
bossyTeacher 5mo ago
X is Y. Z is Y. And Alpha is genuinely Beta. Classic LLM writing style.
driverdan 5mo ago
There's interesting stuff in this writeup but it sure seems like most of it was written by an LLM.
bitwize 5mo ago
You know what the sad bit is? Humans do write exactly like that. That's not even particularly egregious StalkedIn marketroid speak.

matja 5mo ago

The AMD MI250X GPUs are also interesting - 128GB of HBM2E at 3TB/s, sometimes you see them second-hand for under $1k, the catch obviously is that it needs an OAM socket. Never seen an easy way to hook them up to a regular mainboard.

Teknomadix 5mo ago
These are interesting, and offer beefy through put. No point in adapting to a PCI lane thought, stuck behind the slot-bus bottleneck.
Gracana 5mo ago
An additional complication is that MI250Xes are two GPUs in one package, so you need to connect the first and last x16 SERDES groups to the host, otherwise you'll only see one GPU (or it won't work at all, idk). Also, the cheap HPE pulls on eBay need some proprietary HPE magic to work, and I have yet to see anyone figure that out.
plagiarist 5mo ago
Ahh luckily this OAM socket will prevent me from spending money.
- selectively 5mo ago
  [dead]
sonzohan 5mo ago
This person has built a converter for the OAM socket, but it is only confirmed working with NVIDIA cards at the moment (https://www.reddit.com/r/NVIDIA_SXM2PCIE/comments/1d076cn/oam_to_pcie/ https://www.reddit.com/r/NVIDIA_SXM2PCIE/comments/1d076cn/oa...) It fits an MI250X, and the system sees it, but the drivers don't work. They tested an HPE MI250X. There's a rumor on the thread that there are two kinds of MI250X: Ones from HPEs and everyone else's. The HPEs require a special firmware, the normal ones do not. However, the majority of the MI250Xs on the secondhand market are HPE so caveat emptor.

knollimar 5mo ago

A little bit of local copium but neat read. Isn't a rasbpi with 16gb of RAM $300 now?

matja 5mo ago
The latest Raspberry Pi 5 has one 32-bit channel (2x 16-bit subchannels) of LPDDR4X-4267 SDRAM giving 17.1GB/s of bandwidth, 52x less than this GPU. Never mind lacking the CUDA and Tensor cores, so the FP16 performance is 102x less (307 GFLOPS vs 31.4 TFLOPS). So for £200, there's absolutely no comparison for this specific use-case.
- knollimar 5mo ago
  Yeah thats what I'm saying. How is it so cheap????
  - feisuzhu 5mo ago
    V100 GPUs are e-waste.
thejj100100 5mo ago
I don't understand what point you're trying to make here? Are you talking about the price of RAM?

jmyeet 5mo ago

Some context: - In 2017, the v100 was a ~$10,000 GPU. I believe there was a PCI-e version but this is probably so cheap because SXM2 is going to be harder to use; - A 5090 has 1800GB/s of internal memory bandwidth (compared to 900GB/s in the 9 year old GPU). Of course a 5090 is substantially more expensive; - A 5090 has ~21k CUDA cores vs ~5k; - The current $10k NVidia GPU is the RTX 6000 Pro w/ 96GB of VRAM. It has slightly more CUDA cores but it otherwise pretty much just a 5090. This is unsurprising. NVidia uses VRAM for market segmentation. Consider this: in 5-10 years, the trillions spent on AI data centers will likewise be sold for scrap most likely. That's how short the runway is for OpenAI and Anthropic to recover that investment. Anyway, I'm kind of impressed the author managed to get this all to work. I don't think it even would've occurred to me that someone had made an SXM2 adapter, particularly because it's not even used anymore. Like props to whoever did that.

b112 5mo ago
I bet 3 years, but otherwise agree.
echelon 5mo ago
> Consider this: in 5-10 years, the trillions spent on AI data centers will likewise be sold for scrap most likely. That's how short the runway is for OpenAI and Anthropic to recover that investment. Even more interesting: it'll devalue all of SaaS and the entire US tech sector. We might have just shot our most valuable non-AI tech products in the foot.
- wholinator2 5mo ago
  How so? I understand that flooding the market with physical goods will reduce prices and thus profits. But how would that also reduce the nonphysical SAAS stuff?
  - mschuster91 5mo ago
    > But how would that also reduce the nonphysical SAAS stuff? The resulting economic crash will affect everyone, we're (IMHO) looking towards a dotcom-bust level wipeout. And many SaaS and other companies run asset-lean (i.e. they have no server hardware because that's all cloud, no real estate because it's all either wework or conventionally rented), margin-lean (the VC business model requires that, as the basic recipe is to achieve market domination by burning cash) and cash-lean (often enough, it's less than a quarter of expenses on the bank accounts). All that "lean-ness" looks great on an investor's quarterly release sheet: no massive amounts of wealth tied up in assets and no cash sitting around on bank accounts that could be released towards investors as dividends or, if it comes from third parties, costs the company interest... but it prevents resiliency against crises.

mondainx 5mo ago

Great write-up, I've often considered these DC cards for a project and now you've convinced me to pick one up; you describe the price of the unit against what one spends on tokens and that does it for me.

tymscar 5mo ago
Thats why I did it. I think it’s important to put things like that into perspective

casey2 5mo ago

Some resell group is going to have to make this easier. The shear amount of these cards otherwise heading towards the landfill is staggering. That is if Big Tech don't destroy them to prevent model weights from leaking.

eric__cartman 5mo ago
How would destroying the GPUs prevent the model weights from leaking? By the time you get your hands on them the memory is powered off for a long enough time that a cold-boot style attack is impossible.
- sethops1 5mo ago
  Would you bet your trillion dollar company on that? Or would you smash up the garbage [to you] memory chips to be sure.
  - marcosdumay 5mo ago
    It's volatile memory, not flash.
  - tymscar 5mo ago
    I would bet my trillion-dollar company on it because I understand how RAM works.
Alifatisk 5mo ago
> The shear amount of these cards otherwise heading towards the landfill is staggering. The thought of throwing away working cards sounds so bizarre to me. I can't believe companies would dispose them into the landfill like that, it is at least worth giving away for refuse.
- wookmaster 5mo ago
  There’s a long history of corporations doing evil things to ensure their business model succeeds

Teknomadix 5mo ago

Tesla V100 SXM2 16GB is NOT DGX class as the author writes. It's HGX class. The V100 comes in two classes, SXM2 and SXM4, the latter coming with a Max of 80gb on board memory. Typically these are installed 8×A100 80GB SXM4 on an HGX riser, and what that gives you is NVSwitch fabric and 640GB of pooled HBM2e (on package stacked memory /w ~2 TB/s of memory bandwidth). 2u standard rack footprint too.

legitronics 5mo ago
I have no idea what you are trying to say. V100 came as sxm2 and sxm3. And it was 16 and 32gb. HGX is DGX with extra toppings.
- _zoltan_ 5mo ago
  No. DGX is when it's Nvidia's design, HGX when it's a 3rd party design.
_zoltan_ 5mo ago
What on earth are you talking about? Your comment makes no sense. The V100 and A100 are different generations altogether. The V100 does not have 2TB/s.

omarqureshi 5mo ago

Could probably avoid the crazy fan with a waterblock - I've seen a whole kit, v100 + PCIE adapter + block for £235. Yes, you'll have to pay for pump, radiators and radiator fans, but that should really quieten it down

pogue 5mo ago
Someone's already made such a kit as you describe to fit in a consumer PC case and work properly?
- deleted 5mo ago
  [deleted]
- omarqureshi 5mo ago
  https://www.ebay.co.uk/itm/406939340557?var=677143153030&mkevt=1&mkcid=1&mkrid=710-53481-19255-0&campid=5338749390&toolid=20006&_ul=GB&customid=CjwKCAjwuO_QBhAWEiwAIkVhU6ne-qhrP48l-uTWETS88KvgKcphn7XUTPUriwgtA8YHYhG2kEiffhoCAm0QAvD_BwE|0AAAAADihhqWHf8GvoHUZ0mwHZX98FeGvV|CkAKCAjwlerQBhAVEjAAO1hnVkWOFBn0RwIjpOQilhR03KVRGI_V6TeI0pxjEH3-ddHuJ0rf_RA_ogU38eIaAsf0&gclid=CjwKCAjwuO_QBhAWEiwAIkVhU6ne-qhrP48l-uTWETS88KvgKcphn7XUTPUriwgtA8YHYhG2kEiffhoCAm0QAvD_BwE&gbraid=0AAAAADihhqWHf8GvoHUZ0mwHZX98FeGvV&wbraid=CkAKCAjwlerQBhAVEjAAO1hnVkWOFBn0RwIjpOQilhR03KVRGI_V6TeI0pxjEH3-ddHuJ0rf_RA_ogU38eIaAsf0&loc_interest_ms=&loc_physical_ms=9045504&adtype=pla&gad_source=1&gad_campaignid=17335954774&gbraid=0AAAAADihhqWHf8GvoHUZ0mwHZX98FeGvV https://www.ebay.co.uk/itm/406939340557?var=677143153030&mke...
- omarqureshi 5mo ago
  32 GB kit as well https://www.ebay.co.uk/itm/406939344915?_trkparms=amclksrc%3DITM%26aid%3D1110006%26algo%3DHOMESPLICE.SIM%26ao%3D1%26asc%3D303466%26meid%3Dfb09c9c6d3e04fd8b809446bd02877df%26pid%3D101224%26rk%3D2%26rkt%3D5%26sd%3D406939340557%26itm%3D406939344915%26pmt%3D0%26noa%3D1%26pg%3D2332490%26algv%3DSimOrganicCassiniWithToraRecalls%26brand%3DNVIDIA&_trksid=p2332490.c101224.m-1&var=677143082490 https://www.ebay.co.uk/itm/406939344915?_trkparms=amclksrc%3...

deleted 5mo ago

[deleted]

hypfer 5mo ago

[dead]

recursivegirth 5mo ago

> The compute is still real. The VRAM is still real. And the memory bandwidth is where it gets genuinely surprising. Had to stop there. Annoying. I can't stand AI use for writing. It makes any otherwise great article feel so disingenuous.

m0rde 5mo ago
What a difficult world you must live in these days
- peddling-brink 5mo ago
  While I don’t disagree with their sentiment, I’m far more annoyed with it than the AI writing.
  - m0rde 5mo ago
    Yeah. I get that many HN comments are just complaints (heck mine was too and just as negative and shaming). But how bad of a day must you be having to try to shame someone about how they choose to write up an experience they thought was neat. Whatever, free speech and all that. Hope OC's day gets better.
    - gsquaredxc 5mo ago
      It doesn’t read like shaming to me. It’s, in the grand scheme of HN comments, definitely on the more constructive side of the criticism. Maybe it could have been reworded, but I think the author of the post could very easily find it actionable in the future. I too had to stop reading the article at that point, so I think if the author wants more people to read, my advice for them is to just write like themselves. We’ve entered the start of a new Instagram filter age where many people feel they need to have LLMs reword their writing presumably for the same reasons as the original filter age. I share OC’s sentiment of pushing against the recent trend of implicitly shaming people for their individualistic writing styles.
  - qingcharles 5mo ago

mickeyp 5mo ago

Impressive work. But the problem is not the 30 tok/s which is fine for agentic coding and chat. It's prefill; slow prefill kills agentic workloads dead. If you have 100,000 tokens at ~150tok/s per the OP, you're looking at: You have: 100000 / (150/s) You want: hms 11 min + 6.6666667 sec Which is quite a wait indeed.

Aurornis 5mo ago
Most people won’t be dumping 100K tokens into it at once, but I agree that all of the prefill time that adds up during a session becomes a lot to account for. This is also a problem for all of the Mac local LLMs. Macs are a great way to get a lot of high bandwidth memory, but their compute is very far behind current gen dedicated GPUs. Some of the expensive Mac Studio setups allow you to run very large models with usable tokens/s, but you can be waiting a long time for it to get to the point of generating those tokens.
- Tepix 5mo ago
  When you're using OpenCode it's easy to reach 100,000 tokens after a while.
- pyrolistical 5mo ago
  The prefix cache is working properly 100k doesn’t prefill more than once
HarHarVeryFunny 5mo ago
I wonder if this could be usefully mitigated with a combination of prompt (prefix) caching and an agent that let you control what the prompt prefix consisted of. The goal would be to incur that slow prefill once to build the prompt cache, then have subsequent prompts consist of mostly this fixed prefix plus specific instructions. For a language like C++ where modules are split into definition (.h) and implementation (.cpp) parts, one choice of prefix would be all the header files for the project (which aren't likely to change much). More generally the idea would be to have an agent that had cached-prefix reuse as it's primary context management goal. Another possibility, to support caching of files that have since changed, would be for the agent to build the context as a fixed prefix reflecting some or all of the codebase in its start-of-session state, then append any changes to that, with appropriate prompting to only use the latest definition of a function. e.g. Say file A initially contains functions X, Y and Z, then the prompt prefix is built to include X Y Z. If the user then modifies Y -> Y', then just add that to the context, so that the cached prefix is unchanged, giving X Y Z Y'.

whoamii 5mo ago

The real question: did your local LLM write this post?

20wenty 5mo ago
There are many tells aren't there? There was clearly hard human work and experimentation here, but it's a shame the OP let AI do chunks of the writing. Once you see it, it's much harder to take the post seriously.
- iugtmkbdfil834 5mo ago
  I disagree. Not everyone has a good writing style. In those instances I think it is fair to default to llm recommendation. We may be allergic to it, but we saw one formulaic response too many ( though admittedly it does raise a question of whether HN was the intended audience for it ). In any event, not all of us have a unique writing style worth preserving just like not all of us can write clear and clean code. Just saying.
  - unshavedyak 5mo ago
    I really wish it was more common to use AI for augmenting than authoring. Eg i find coding with LLMs neat when you primarily "talk" to it through code, by filling out structs, funcs, fields, etc - where it would use your changes as the template and then to work to effectively autocomplete the gaps. The more you iteratively write the less it fills in, but also the less it deviates from your intent, design, etc. I feel like writing could use a similar harness, where it attempts to minimally reword the authors sentences, perhaps just tweaking grammar, spelling, etc. In the coding example i think the human code would be near unchangeable, the LLM would pivot around it - but in the writing example i think the human writing would have to be more mutable. I imagine it would be a configurable setting. I've not really seen a system which focuses on this human<->LLM look, but it feels interesting to me.
    - iugtmkbdfil834 5mo ago
      In a sense, there is a clear market for it ( people want 'authentic' experience ). I can kinda understand it. I want pure linux experience without systemd, but I recognize that in the current ecosystem, it comes at a cost. So the language harness makes sense to me, but corps are already cracking down on token use ( and such a harness would likely only add to the cost ). The other question is whether the people, who could benefit it would even recognize it as a problem though.

bob1029 5mo ago

> And yes, if you want the absolute best, Opus 4.8 exists. It also costs more per 20 minutes of heavy use than I paid for this entire GPU and adapter setup combined. But the gap is shockingly small. I don't think this is a fair characterization of the situation. I use frontier models via API pre-paid tokens every single day, and I can barely rack up $100 per month. The fact that we figured out how to burn double this in 20 minutes is impressive, but I don't think it reflects the reality that many are experiencing right now. There are some exceptionally gluttonous approaches to harnessing LLMs that I think are serving as convenient straw men in these discussions. Paying for the API will almost always be more economical than self-hosting equivalent infrastructure. I am not against self-hosting, but the article suggests a primarily economic motivation for this effort. If you are consuming fewer than 10^9 tokens per month, I really don't think it's worth your time to try and compete with the hyperscalars. Most of the money is to be found in the integration of this technology with existing businesses.

vidarh 5mo ago
I use hosted providers myself, but I can churn through $100 worth of tokens in half a day even with cheap models like Deepseek easily. If someone's use is as light as yours, then sure - grab a subscription and you'll save far more. For higher use it will come down to how cheap your electricity is whether it is worth offloading at least some of it (for me it's not, FWIW)
- iJohnDoe 5mo ago
  Could you share a bit about what you’re working on or what type of projects require that much usage? Is it hobby, production, revenue generating?
  - vidarh 5mo ago
    A mix. I have hobby projects that churn through that much when I don't need the tokens for others things. I also have projects for clients that easily consumes those levels. As well as a stealth-ish potential startup. Currently I'm at 4 different subscriptions + more than I'd like in spend via OpenRouter... What multiplies it very quickly is when you start feeding them with test suites and "Ralph loops" that run until the test suites pass, or complex chains with lots of sub-agents being triggered. If you're sitting there watching everything, it will be hard to burn all that much even if you're running multiple things in paralle.

ewy1 5mo ago

despite gaming being used in the title, it is not mentioned in the article, but i'm curious how this performs. i've ran some multi vendor frankenstein setups before and sometimes it even works, so i'm curious to hear your experience with it.

pogue 5mo ago

But could you game with the GPU? Or is that purely a drivers issue?

mcraiha 5mo ago
I assume you can game with it if you use modded drivers. At least with CMP 50HX that is possible https://videocardz.com/newz/nvidia-cmp-50hx-turned-into-gaming-gpu-shows-lackluster-performance-due-to-limited-pcie-lanes https://videocardz.com/newz/nvidia-cmp-50hx-turned-into-gami...

abejfehr 5mo ago

Based on the title I was really hoping to see how this was used for gaming, but they just ran an LLM on it

axpy906 5mo ago
Same. With no new NVIDIA gaming GPUs this year, seems like an interesting problem to solve.
mschuster91 5mo ago
I don't think that is even possible, every piece of silicon on that chip that is required to do gaming is ripped out in favor of more compute cores.
darkwater 5mo ago
They said in the beginning that it doesn't even have a video out, so you cannot do gaming.
- yjftsjthsd-h 5mo ago
  I thought you could run games by rendering on one GPU and outputting on another? Usually comes up with dual iGPU/dGPU setups, but could work here
- toast0 5mo ago
  I've seen things where you have multiple video cards and can use one gpu to render to a framebuffer which is transferred to the other video card to output. I'm sure it adds latency, and it's probably unsupported... But no output doesn't mean can't do gaming... It just means gaming will be iffy. There's some virtualized desktop server stuff too. Run a bunch of desktop sessions on a beefy computer and send a video stream to desktop players. With the right codec settings, the latency is probably ok for many games.
- hakfoo 5mo ago
  I'm actually surprised there hasn't been a dedicated effort to support display offload to, say, the CPU's iGPU. I'm sure manufacturers would love saving a dollar per card, and OEMs would appreciate eliminating the support calls from "I just bought a new $2500 gaming PC and no video" because they plugged the monitor into the iGPU instead of dGPU.

wg0 5mo ago

Wait a few years, everyone will be able to put one at half the price.

KnuthIsGod 5mo ago

AI written posts will kill HN.

tymscar 5mo ago
AI didnt edit a single word of this post.

axpy906 5mo ago

Wow. V100. That brings back memories. Way to go.

gtirloni 5mo ago

> The compute is still real. The VRAM is still real. And the memory bandwidth is where it gets genuinely surprising. sigh

krackers 5mo ago
Usually that cloying pattern is reserved for "emotional" contexts to validate the user ("your struggles are real [despite others thinking it's in your head]"). Here it doesn't even make sense, of course the VRAM is real. Is it going to tell me that my keyboard is real next? I wonder if this was generated with the local model, this seems to be a case where it memorized the style but not the meaning and intent.

viseyth 5mo ago

Volta (and Pascal, which I'm using) should still be supported with driver 580 as long as you don't use the open modules, and you can use up to cuda 12.9 and cudnn 9.10.2. No need to limit yourself to an old kernel.

markus92 5mo ago
It is. We still run quite a few of them in prod and with 580 drivers they run just fine. Very useful GPUs still.

segmondy 5mo ago

The most interesting and perhaps useful for most would be how they control the fan. If you are thinking of doing this, you really want to get those fans under control, they are loud. For anyone thinking of these, v100s idle super high! 25-35watt with nothing loaded and easily 50w when a model is loaded.

00dazzle 5mo ago

That's the same price per VRAM GB as an arc pro B70

tymscar 5mo ago
But with miles better support, thats why I went this route. Cuda is hard to beat

jonhohle 5mo ago

I was just looking into this and was worried about the fan setup. Interesting that he was able to solve it with good results. In case anyone is interested, I’m using PCIE passthrough on a FreeBSD host to a Linux guest with an older Pascal card. It’s worked great and I’ve been thinking about putting a nicer card in there. The SXM route seems great, but I’ve been burned (almost literally because of the heat) by DC components before.

sonzohan 5mo ago

I also recently decided to buy a datacenter GPU and slap it into a system. Some notes from my experience that the author doesn't mention in their article: Decommissioned NVIDIA V100s and AMD MI50s are fairly cheap, $200 for 16gb and $400-500 for 32gb, for local experimentation. They are also very old. There's an enthusiast community keeping these two cards alive and working with current platforms and models. Nitpick, but the V100 doesn't support bfloat16. The performance hit is not a big deal if you're fiddling with local models, but the card is on it's way out in terms of hardware features. The MI50 does support bf16, but not the current edition of AMD ROCm. Vulkan support is good and the MI50 works with most major platforms (llama.cpp, vllm, etc.), but it's not without some pain points like manual recompilation. Fortunately the open source community has already paid most of your way. The cooling requirements for these cards cannot be understated. A consumer grade GPU may throttle if in a small case without additional fans, but if given the same treatment a datacenter GPU will overheat itself idling. You will need to buy, at least, a bunch of decent 120mm fans to prevent this or invest in some water cooling. I ultimately went with an AMD MI100 32GB ($950). I'm an AMD fan, current ROCm editions support it, and it was low-fuss to get things working. I'm debating getting a second so I can try out bigger models like qwen3-coder-next.

Silagi 5mo ago
Did you consider the R9700 or B70 when you went for the MI100? If so, what made you choose the MI100? I've been playing with picking up a card in this class but haven't been able to justify it when running the Qwen3.6 MOE model on a 6800xt is tolerable for the type of projects I've been willing to point local AI at.
- sonzohan 5mo ago
  I looked at those, the Arc 1100, the w6800, MI50, MI60, v100, v620, and basically anything with 32gb of RAM: 1. I wanted an AMD card. 2. I have an RTX 3090 that's been fun to play with, but I want to get back to using it for gaming. 3. I was looking for between 30-60 tokens/second in terms of performance on the beefier models I want to run. Looking at stock Qwen3 32B the benchmarks reported about 41 tokens/second for MI100. w6800 was 18, MI50 & MI60 could do 60s but had a lot of compromises/special things to achieve that. 4. I used FitMyLLM for some spec-based comparisons (https://www.fitmyllm.com/ https://www.fitmyllm.com/). The MI100 is roughly double the performance on Qwen 3.5 35B A3B Q5_K_M to the R9700 (462 token/s prefill vs 239 tokens/s, 217 tokens/s vs 118 token/s for inference) 5. I was willing to throw up to $1k at a GPU; I really wanted to throw closer to $650. To be honest, if money was no objection I would've sprung for a MI210. I also considered the MI250 as they showed up for $1250-1400 with a whopping 128GB, but the PCIE converters for that form factor don't have working AMD drivers yet.

melonpan7 5mo ago

Great value for money if you have the time for tinkering and getting the compatibility to work.

DANmode 5mo ago
and it only gets easier and more defined from here.
tymscar 5mo ago
If you use NixOS, you don’t need to do any of that anymore. A single command in the CLI, and you have everything I do.

mettamage 5mo ago

> The way it works is that a vision encoder (similar to what ChatGPT and Claude use) takes image pixels and translates them into the LLM’s token embedding space. The model does not “see” the image the way a human does. Instead, the vision encoder compresses the image into a sequence of vectors that live in the same mathematical space as text tokens. The LLM then processes those vectors as if they were just another sequence of tokens. Could you also do this for music and specifically sound synthesis? It would be awesome to vibe synthesize sounds and then see the VSTi parameters surrounding it.

Ey7NFZ3P0nzAe 5mo ago
I don't think so. Cramming new senses into the latent space of the model is one thing, but having a model output tokens that can be detokenized into sound is completely different and requires a very different type of data.
- LearnYouALisp 5mo ago
  What do you mean? Why not? We can already FFT sound into "words", so why not have some kind of dictionary to an arbitrary level of precision/fidelity?
  - Ey7NFZ3P0nzAe 5mo ago
    Sound frequency is continuous : "between sound of A and O" means something, but "between the letter A and F" means nothing.

jeffrallen 5mo ago

Super interesting. I use data center GPUs at work, but I didn't know anything about this stuff. I also use Qwen 3.7 27b at work and I agree with the author: it is perfectly capable of the jobs I give it.

peibye 5mo ago

All that work just to write an ai blog post. This is a cool topic but I just can’t deal with the aiisms.

drumhead 5mo ago

The first dual core processor I built a machine with was an Opteron. It was a nice piece of hardware.

LearnYouALisp 5mo ago
xw9300 fan

suralind 5mo ago

This is great! I've been trying to get into local models for a while as I share the sentiment that local models will eventually be so good that there won't be a need to use frontier models for most coding tasks (perhaps that's already true today?). I have zero experience building computers - where would I even start? I mean, aside from the things already well documented and mentioned in the blog post.

tymscar 5mo ago
Building computers is very easy. I would suggest watching a YouTube video to get the general gist, and then once you buy the parts, just Google for whatever doesn’t go well. I built my first Pentium 4 one when I was like six, so I’m sure someone much older that’s into tech can do it without an issue. There are also tons of Discord communities that are willing to help you live if you encounter any issues.

binyu 5mo ago

The V100 and the 4090 are based on vastly different architectures, the former uses the older Volta while the latter uses Ada. Last I checked you cannot meaningfully combine them. The 3090 is better than the V100, just get two 3090 and a NVLink.

tymscar 5mo ago
Well I did in fact meaningfully combined them without an issue, that was the whole point of the blogpost.
- binyu 5mo ago
  Yes but it creates a bottleneck that negates the benefit of using multiple cards that way. Look into it. Cheers
  - tymscar 5mo ago
    Well it doesn’t matter because the bottleneck here is actually quite small for me. The issue is vram. If anything the bottleneck is my 4080.
    - binyu 5mo ago
      Gotcha, I am not saying your setup is inherently wrong or useless. I am glad it works for your use cases. Godspeed
      - tymscar 5mo ago
        I think its a very fair thing you have flagged!
cthalupa 5mo ago
You can split tensors across an AMD GPU and Nvidia GPU - different architectures are not an issue. People run LLMs across some pretty crazy setups.

QuantumNoodle 5mo ago

This is very cool. Can anyone point me to any resources for setups that can ron decent inferencing at home?

mg794613 5mo ago

Very interesting, but slight note on the fact that writer sort of forgets to calculate in the price of the 4080 it works together with. He spend 200 to upgrade his existing setup, great nonetheless. But not "32gb gram for only 200 bucks"

tymscar 5mo ago
I did mention that in my case I had the 4080, but you do not need the 4080 whatsoever. You can run the same model with less context on a single £200 V100 or with the same amount of context on two of these.

rbanffy 5mo ago

This lack of driver support for hardware that’s still functional and available in reasonable quantities is annoying. I was very close to buying a retired POWER7+ server with an ungodly amount of memory, but decided being unable to run a modern Linux kernel would be more work than I wanted to have. Modern kernels need POWER8 and above.

rbanffy 5mo ago

neutrinobro 5mo ago

Very nice to see this older hardware getting repurposed. I have been running 2x Tesla V100s in a dual-core supermicro X10DRU-i server. With qwen3.6-27B-mtp I get about 35-40tok/s for inference for moderate context sizes (<128k), and have run long running agent tasks on it which consume 100s of millions of tokens (>$100s if I had to pay claude API costs). However, the main purpose that I have to for these cards is for scientific compute, the FP64 performance (7+ TFLOPS!) is fantastic given their age, and not something you can get on even the latest consumer grade cards since Nvidia nerfed their performance after Kepler. The server lives in the basement though...it is freaking loud!

Ey7NFZ3P0nzAe 5mo ago

While we're at it, has anyone any tips on turning a consumer RTX 3090 into a blower type format for putting inside a server rack? Any kind of workaround like this.

j4k0bfr 5mo ago

Thank you for writing this! And sharing the NixOS config. I've been looking for an excuse to get into Nix for ages, but this might finally do it. P.S. Man the AI writing complaints on this thread are quite upsetting. OP is handling it a lot better than I would, lol.

AI2070 5mo ago

now design a shroud and blower