6 ms·

Every single day, three things are becoming more and more clear: (1) OpenAI & Anthropic are absolutely cooked; it's obvious they have no moat (2) Local

• kcb 6mo ago
  What benefit is there to dropping $50k on GPUs to run this personally besides being a cool enthusiast project?
  • blizdiddy 6mo ago
    Is it so hard to project out a couple product cycles? Computers get better. We’ve gone from $50k workstation to commodity hardware before several times
    • kcb 6mo ago
      Subscription services get all the same benefits from computer hardware getting better. But actually due to scale, batching, resource utilization, they'll always be able to take more advantage of that.
  • deminature 6mo ago
    Intel has just released a high VRAM card which allows you to have 128GB of VRAM for $4k. The prices are dropping rapidly. The local models aren't adapted to work on this setup yet, so performance is disappointing. But highly capable local models are becoming increasingly realistic. https://www.youtube.com/watch?v=RcIWhm16ouQ https://www.youtube.com/watch?v=RcIWhm16ouQ
    • kcb 6mo ago
      That's 4 32GB GPUs with 600GB/s bw each. This model is not running on that scale GPUs. I think something like 96GB RTX PRO 6000 Blackwells would be the minimum to run a model of this size with performance in the range of subscription models.
      • acchow 6mo ago
        > I think something like 96GB RTX PRO 6000 Blackwells would be the minimum to run a model of this size with performance in the range of subscription models. GLM 5.1 has 754B parameters tho. And you still need RAM for context too. You'll want much more than 96GB ram.
  • marcus_holmes 6mo ago
    Why would anyone need more than 640Kb of memory?
    • kcb 6mo ago
      Exactly the point though. In the 640KB days there was no subscription to ever increasing compute resources as an alternative.
      • marcus_holmes 6mo ago
        Well, there kinda was - most computing then was done on mainframes. Personal / Micro computers were seen as a hobby or toy that didn't need any "serious" amounts of memory. And then they ate the world and mainframes became sidelined into a specific niche only used by large institutions because legacy. I can totally see the same happening here; on-device LLMs are a toy, and then they eat the world and everyone has their own personal LLM running on their own device and the cloud LLMs are a niche used by large institutions.
        • kcb 6mo ago
          The difference is computers post text terminal are latency and throughput dependent to the user. LLMs are not particularly.
          • marcus_holmes 6mo ago
            Sorry, I don't understand that comment. Can you clarify, please?
            • kcb 6mo ago
              My point is LLMs aren't more usable if the hardware is in your room versus a few states away. Personal computers still to this day aren't great when the hardware is fully remote.
              • 6mo ago
  • CamperBob2 6mo ago
    It will run exactly the same tomorrow, and the next day, and the day after that, and 10 years from now. It will be just as smart as the day you downloaded the weights. It won't stop working, exhaust your token quota, or get any worse. That's a valuable guarantee. So valuable, in fact, that you won't get it from Anthropic, OpenAI, or Google at any price.
    • kcb 6mo ago
      That's why we all still use our e machines its never obsolete PCs. Works just the same it did 20 years ago, though probably not because I've never heard of hardware that's guaranteed not to fail.
  • fwipsy 6mo ago
    Agree directionally but you don't need $50k. $5k is plenty, $2-3k arguably the sweet spot.
    • unlikelytomato 6mo ago
      as a local LLM novice, do you have any recommended reading to bootstrap me on selecting hardware? It has been quite confusing bring a latecomer to this game. Googling yields me a lot of outdated info.
      • fwipsy 6mo ago
        First answer: If you haven't, give it a shot on whatever you already have. MoE models like Qwen3 and GPT-OSS are good on low-end hardware. My RTX 4060 can run qwen3:30b at a comfortable reading pace even though 2/3 of it spills over into system RAM. Even on an 8-year-old tiny PC with 32gb it's still usable. Second answer: ask an AI, but prices have risen dramatically since their training cutoff, so be sure to get them to check current prices. Third answer: I'm not an expert by a long shot, but I like building my own PCs. If I were to upgrade, I would buy one of these: Framework desktop with 128gb for $3k or mainboard-only for $2700 (could just swap it into my gaming PC.) Or any other Strix Halo (ryzen AI 385 and above) mini PC with 64/96/128gb; more is better of course. Most integrated GPUs are constrained by memory bandwidth. Strix Halo has a wider memory bus and so it's a good way to get lots of high-bandwidth shared system/video RAM for relatively cheap. 380=40%; 385=80%; 395=100% GPU power. I was also considering doing a much hackier build with 2x Tesla P100s (16gb HBM2 each for about $90 each) in a precision 5820 (cheap with lots of space and power for GPUs.) Total about $500 for 32gb HBM2+32gb system RAM but it's all 10-year-old used parts, need to DIY fan setup for the GPUs, and software support is very spotty. Definitely a tinker project; here there be dragons.
        • terbo 6mo ago
          Agree on the framework, last week you could get a strix halo for $2700 shipped now it's over $3500, find a deal on a NVME and the framework with the noctua is probably going to be the quietest, some of them are pretty loud and hot. I run qwen 122b with Claude code and nanoclaw, it's pretty decent but this stuff is nowhere prime time ready, but super fun to tinker with. I have to keep updating drivers and see speed increases and stability being worked on. I can even run much larger models with llama.cpp (--fit on) like qwen 397b and I suppose any larger model like GLM, it's slow but smart.
• jimmaswell 6mo ago
  No killer product? Coding assistants and LLM's in general are the single most awe-inspiring achievement of humanity in my lifetime, technological or otherwise. They've already massively improved my and others' lives and they're only going to get better. If pre and post industrial revolution used to be the major binary delineation of our history, I'm fairly confident it will soon be seen as pre and post AI instead.
  • bitexploder 6mo ago
    No killer products... just robots that can do vulnerability analysis at the level of a decent security engineer and write code without tiring.
  • allan_s 6mo ago
    I've also been using the LLM in Posthog and it has been impressive. I need to check if I can also plug a MCP/Skill to my actual claude code so that I can cross reference the data from my other data source (stripe, local database, access logs etc.) for in depth analysis
    • chris_ivester 6mo ago
      This might be up your alley - have Posthog and a ton of other SaaS tools connected so you can run analysis across quant/qualitative data sources: https://dialog.tools https://dialog.tools
  • pdntspa 6mo ago
    I know right? 8-year-old me dreamed of being able to articulate software to a computer without having to write code. It (along with the original Stable Diffusion) are Definitely one of the coolest inventions to ever come along in my lifetime
  • zozbot234 6mo ago
    Coding assistants are currently quite hard to run locally with anything like SOTA abilities. Support in the most popular local inference frameworks is still extremely half-baked (e.g. no seamless offload for larger-than-RAM models; no support for tensor-parallel inference across multiple GPUs, or multiple interconnected machines) and until that improves reliably it's hard to propose spending money on uber-expensive hardware one might be unable to use effectively.
• eldenring 6mo ago
  I don't see how its possible to think this. AI coding assistants are some of the most useful technologies ever created, and model quality is by far the most important thing, so I doesn't make sense why local inference would be the path forward unless something fundamentally changes about hardware.
  • sunir 6mo ago
    The hardware will change. We know that.
• anon291 6mo ago
  (3) is simply a lie spread by engineers who have no other context. I manage some real estate (mid-term rentals) and everyone I know has switched over to AI robo-handlers to do the contact at this point. It's almost a passive investment at this point. Some can even handle interfacing with contractors and service requests for you. Revolutionized the field in my opinion.
• mgfist 6mo ago
  Posted this after mythos came out? The hutzpah
• grafmax 6mo ago
  > no moat I'd like to think the superior product wins. But Windows still thrives despite widespread Linux availability. I think sometimes we can underestimate the resilience of the tech oligopolies, particularly when they're VC-funded.
  • jjfoooo4 6mo ago
    VC can spend all the money in the world and it won't matter if the cost of switching providers is effectively zero. If I want to switch from Windows to Linux, I have to reconsider a whole variety of applications, learn a different UX, migrate data, all sorts of annoyances. When I switch between Codex and Claude Code, there is literally no difference in how I interact with them. They and a number of other competitors are drop in replacements for each other.
  • AlienRobot 6mo ago
    >I'd like to think the superior product wins. But Windows still thrives despite widespread Linux availability. That's because by most metrics Linux is inferior is Windows.
• hodgehog11 6mo ago
  (1) is absolutely not true if you actually use these models on a regular basis and include Google in here too. The difference in reliability beyond basic tasks is night and day. Their reward function is just so much better, and there are many nuanced reasons for this. (2) is probably true but with caveats. Top-tier models will never run on desktop machines, but companies should (and do) host their own models. The future is open-weight though, that much is for sure. (3) This is so ignorant that others have already responded to it. Look outside of your own bubble, please.
  • neonstatic 6mo ago
    > Top-tier models will never run on desktop machines Sorry, but you don't know that
    • Yiin 6mo ago
      I mean it's not hard to understand that if good model can run on consumer hardware, even better models can run in data centers
      • neonstatic 6mo ago
        Larger, yes, absolutely. Better? Right now it seems that bigger is better, but if we are thinking about long term future, it's not obvious that there isn't a point of diminishing returns with regards to size. I can also imagine a breakthrough, where models become much smaller, with the same or better capabilities as the current, very large ones.
        • hodgehog11 6mo ago
          You are always going to get the same scaling laws in model size regardless of what else you do, so the same degree of improvement seen now relative to the smaller models will be achievable in the future. Yes, small models may be on par with previous generation large models, but the same is true for processors and you don't see supercomputers going away. It's the same principle.
• neonstatic 6mo ago
  The model is the killer product
• fwipsy 6mo ago
  No moat: yes. Cooked: no. It's a race. Why assume they're going to lose? It relies on (2) which is only true if AI usefulness plateaus at some level of compute. That's a huge claim to be making at this stage. (3) AI has lots of killer products already. The big one is filling in moats. Unrealized potential though for sure.
• IncreasePosts 6mo ago
  How good would open source models be if they couldn't distill higher quality private models?
• AlienRobot 6mo ago
  I was trying to use Claude.ai today to learn how to do hexagonal geometry. Every time I asked a question it generated an interactive geometry graph on the fly in Javascript. Sometimes it spent minutes compiling and testing code on the server so it could make sure it was correct. I was really impressed. Anyway I couldn't really learn anything since when the code didn't work I wasn't sure if I had ported it wrong or the AI did it wrong, so I ended up learning how to calculate SDF and pixel to hex grid from tutorials I found on google instead.
  • jurschreuder 6mo ago
    This is also my exact experience
• bottlepalm 6mo ago
  This has got to be bait.. 1) OpenAI and Anthropic are killing it, and continue to do so, their coding tools are unmatched for professionals. 2) Local models don't hold a candle to SOTA models and there's nothing on the horizon that indicates that consumers will be able to run anything close to what you can get in a data center. 3) Coding is a killer product, OpenAI and Anthropic are raking in the cash. The top 3 apps are apps in the app store are AI. Everyone who knows anything is using AI, every day, across the economy.
  • svcrunch 6mo ago
    The grandparent is definitely wrong on (3). Yes, coding is a killer product, I agree with you. On (2), I agree with you for local models. BUT, there are also the open source Chinese models accessible via open-router. Your argument ("don't hold a candle to SOTA models") does not hold if the comparison is between those. On (1), I agree more with the grandparent than with your assessment. Yes, OpenAI and Anthropic are killing it for now, but the time horizon is very short. I use codex and claude daily, but it's also clear to me that open source is catching up quickly, both w.r.t. the models and the agentic harnesses.
    • itake 6mo ago
      > the open source Chinese models accessible via open-router And? They aren't as good as SOTA models. Even the SOTA model provider's small models aren't worth using for many of my coding tasks.
      • DeathArrow 6mo ago
        In my limited experience with it, GLM 5.1 is on par with Opus 4.6.
        • naasking 6mo ago
          I used GLM5 quite a bit, and I'd say it was maybe on par with Sonnet for most simple to medium tasks. Definitely not Opus though. Didn't test super long context tasks, and that's where I would expect it to break down. A recent study on software maintainability still showed Sonnet and Opus were peerless on that metric, although GLM series of models has been making impressive gains.
• DeathArrow 6mo ago
  >(1) OpenAI & Anthropic are absolutely cooked; it's obvious they have no moat I think big corporations will continue to use them no matter how cheap and good other models are. There's a saying: nobody was fired for buying IBM.
• Glaklloo 6mo ago
  Google doesn't release Gemma 4 if Gemini is similiar good. We probably talk abuot a year of progress diffeerence. Its also still quite expensive for an avg person to consume any of it. Either due to hardware invest, energy cost or API cost. Also professionally I don't think anyone will really spend a little bit less money of having the 3th quality model running if they can run the best model. I'm happy that we reach levels were this becomes an alternative if you value open and control though.
• deleted 6mo ago
  [deleted]