5 ms·

With the business model for API based LLMs looking iffy at best it seems like we’re heading back to the “server under your desk” era of IT again.

• cube00 2mo ago
  Considering how all the big players are playing fast [1] and loose [2] with limits, billing [3] and adding undisclosed changes that burn your tokens on autopilot [4], it can't happen soon enough. [1]: Limits may change without notice, including due to capacity constraints. - https://support.google.com/gemini/answer/16275805?sjid=14713165163333653664-NC#zippy=%2Cusage-limit-changes:~:text=Limits%20may%20change%20without%20notice%2C%20including%20due%20to%20capacity%20constraints https://support.google.com/gemini/answer/16275805?sjid=14713.... [2]: "standard limits" are never defined - https://support.google.com/gemini/answer/16275805?sjid=14713165163333653664-NC#:~:text=an%20AI%20plan-,Standard%20limits,-AI%20Plus https://support.google.com/gemini/answer/16275805?sjid=14713... [3]: https://tobyonfitnesstech.com/blog/anthropic-refund-scam/ https://tobyonfitnesstech.com/blog/anthropic-refund-scam/ [4]: https://news.ycombinator.com/item?id=48947776 https://news.ycombinator.com/item?id=48947776
  • xscott 2mo ago
    Not to mention all the other ways they can screw you: - Middle of the day, servers busy? Swap to Sonnet while pretending it's still Opus. Many people won't notice, and nobody can prove anything if they suspect. - Middle of the night, server load is light? Put it into extra thinky mode so it burns more tokens to ramp up the bills. Flip the switch where it gets really pedantic about writing lots of extra test cases and verifying against documentation. - Demand increases, but don't feel like running more hardware? Switch to low bit quants, but have a monitor model swap back to quality if it can tell you're running a benchmark. Assuming model capability plateaus (I think it will), token providers will be in a race to the bottom to maximize profits at the expense of quality that's very difficult to measure.
    • dannyw 2mo ago
      These kind of tricks will completely break API customers and be super visible, since most companies deploying API at scale have ample telemetry, evals, etc. Although, selectively applying it to consumer subs is probably beyond likely at this point.
    • htrp 2mo ago
      Ugh... didn't think about extra thinky mode in the middle of the night. So many ways for enshittification here.
    • mister_mort 2mo ago
      It all sounds like having to rely on a dodgy housing contractor that wants to steal from you, take shortcuts AND choose the gold-plated options from their supplier friends, and will start doing this the minute you are not on site supervising. You don't do it yourself (because the contractor is faster and stronger than you in many ways) but you can't leave, so you're stuck on the worksite just watching them.
      • xscott 2mo ago
        It's worse though, because you can't really watch them at all. It's very difficult to get quantitative numbers for quality. Even within the same model family, same tokenizer, and complete control over the weights and logits, perplexity and KL-divergence isn't really what you want. Now put it behind an HTTP endpoint, and it's just opaque. I've seen local models recognize when the task I'm asking them for is likely to be an artificial benchmark. And any smart company is going to use lightweight models to monitor your sessions. If their sentiment analysis suspects you're close to cancelling, they'll up the knob for a few days until you calm down. Or worse, their accounting tells them that you're getting too much value from your fixed price subscription, so they turn the knob down to encourage you to cancel. In the short term, the "frontier" models are too good to ignore. But if (when?) that plateaus, I don't see how anyone could trust a non-local model. When you pay an ISP to serve your web site, you can tell if they over-compress your images to save storage and bandwidth. With LLMs, it's just JSON with more errors and pointing to the fine print that models are not deterministic.
  • deleted 2mo ago
    [deleted]
• skohan 2mo ago
  I've been coding using the LLM server in my living room for the past few weeks, and I haven't had this much fun with tech for ages
  • exe34 2mo ago
    Can I ask, do you feel the pain of the level of abstraction? I haven't tried local in a few months, but last time I tried, I felt like I was directing a coding exercise - whereas with a frontier model, it feels more like directing a product building. "I need this feature", vs "write code to do this in this file".
    • rufasterisco 2mo ago
      Some people like it better when they direct the solution because they walk away with a better understanding of it. This has emotional/psychological aspects (it feels less like LLMs are replacing you), as well as practical ones (overall complexity is bounded by what the dev brain can understand/grasp). A dev work becomes more and more about reliability, signing off safe software with a litmus test: “I will be on to handle this code failure as if I had written it”. All the above points towards keeping tight control over some level of abstractions and delegating others.
      • exe34 2mo ago
        I find it depends at what stage I'm at with the idea - sometimes I don't want to understand it until it works, because I've wasted enough life on things that didn't do what was promised. But once I know the idea is feasible, yes I would prefer to understand the code at some level.
    • skohan 2mo ago
      > write code to do this in this file I haven't had to micromanage to this level. I usually start with a spec for a feature, which will be as detailed as I am opinionated about the feature. But it's usually on the level of a high-level context, plus some key implementation details (technology choices, key requirements, maybe an interface/API specification to 80% detail), and then the project already has high-level policies documented about e.g. how to structure files within the project. Then I do a planning phase, task breakdown, and implementation of subtasks all within the model. I do read through it, but mostly the quality is good and I might make a couple notes. Then I do a review phase, which usually picks up a couple things. I'm moving towards less manual review of results and more automation as I learn what I can and can't trust the model with. There's definitely a capability gap vs. larger models, but honestly I kind of prefer this workflow, as I stay more in touch with how the codebase is structured. And it's great to be able to experiment as much as I want without worrying about how many tokens I'm burning or how close I am to a usage limit.
• Der_Einzige 2mo ago
  With how expensive consumer hardware is and will continue getting (due to LLM demand), good luck getting a "server under your desk" for something less than an arm, leg, and first born. Until A100 prices are reliably under 1.70$ an hour, there is no GPU/AI bubble and Michael Burry doesn't know anything about GPUs.
  • xscott 2mo ago
    There are lots of points in a spectrum of choices. DGX Sparks, Strix Halos, and the surviving Mac Studios can easily run these 30B class models, just not as fast. So maybe just the leg, but you can keep the arm and first born. And super noteworthy is that a 27B model (Qwen 3.6 27B) from this year is a huge improvement over a 120B model (gpt-oss:120b) from last year. The goal posts are moving, but at some point "good enough" is good enough for the kind programming I like to do.
• drob518 2mo ago
  That’s part of it. There’s also just a natural back-and-forth between what I call “time sharing” and “personal.” When the thing you want is expensive, you share it remotely, but as soon as costs fall, everyone wants it under their desk.
• staplor 2mo ago
  What do you mean iffy? The major AI labs are gross profitable when selling access to inference. In addition, the best models have trillions of parameters and are most efficiently served on large, expensive clusters and served to many concurrent users.
  • wolttam 2mo ago
    They make money on each token when you look at the electricity and interconnect fees, but no, I don’t think they’ve turned a profit on their Capex, even a little bit
  • claytongulick 2mo ago
    > The major AI labs are gross profitable when selling access to inference. Do you have a good source for this?
    • NorwegianDude 2mo ago
      Well, I can run some models that are better than some of the weaker and cheaper Anthropic models locally, like Haiku 4.5, and solve tasks that would cost ~4500$ every day in tokens, so yeah, they are definitely extremely profitable on inference.
      • claytongulick 2mo ago
        Are you also paying 8,000 employees [1] and funding massive infrastructure [2]? [1] https://www.makerstations.io/openai-employee-statistics/ https://www.makerstations.io/openai-employee-statistics/ [2] https://www.wheresyoured.at/oai_docs/ https://www.wheresyoured.at/oai_docs/
  • cmiles8 2mo ago
    That’s like saying an apartment building is “profitable” because the rent covers utilities while ignoring the real cost which is the mortgage on the capital cost of the building. It’s funky math and a good way to quickly go bankrupt.
• gdhkgdhkvff 2mo ago
  Why do you say API based llms looking iffy at best? Do you just mean current profitability due to market pressures from some companies’ subsidized investor money? Surely, even if you’re just using open weights models, it should theoretically be cheaper to use them in a highly optimized cloud architecture(even with vendor markups) rather than each person serving their own models from much less efficient (and more importantly, much less consistent volume) self-owned “server under your desk”?
  • aqme28 2mo ago
    LLMs are becoming commoditized, which means the margins are trending to zero. It's a lot less exciting to spend another trillion on a new model if you can barely make any profit. Meta getting out of the game might be the smarter move.
    • lukeschlather 2mo ago
      Companies are going pretty quiet about costs, but I see no reason to believe the cost to train a model exceeds $5B. Moonshot AI's entire funding is like $5B, their $300M revenue is negligible but not nothing. Renting the compute to train a model like Kimi K3 cannot possibly exceed $5B and is probably under $1B. It's probably at least $100M, but also plausibly not. I don't think it's likely Meta would be giving away models for free if the compute cost to train them was in the billions. This 30B parameter is tiny, that's not billions of dollars, that's likely millions, maybe even less.
• lostmsu 2mo ago
  This release is not a meaningful improvement in any metric over 5 months old Qwen 3.6. DS v4 Flash update maybe, but it is too big for typical Joe's desktop.
• Aurornis 2mo ago
  > at best it seems like we’re heading back to the “server under your desk” era of IT again Maybe in the very long term. If companies go local, the efficient model is to buy some big hardware to share among developers. I run local models. Even with 128GB unified memory systems or a 5090 or RTX 6000, the generation speeds X model quality X context length is still far behind what I get from my SOTA model subscriptions. I also pay a lot more for the locally generated tokens in electricity and hardware costs. I'm also limited in parallel requests to the local box. The list goes on. I really like running local models, but for any given point in time it's more efficient to have a big central box aggregating requests and churning through them. So maybe companies buy $300K servers and try to split it among 30 users instead of trying to buy 30 x $10K boxes. More likely, they rent time on cloud servers by the month so they can adapt the hardware when new models come out with new requirements. Then some day in the distant future when hardware is cheap and plentiful again, it might make sense for us to go back to individual boxes under the desk.
• samtp 2mo ago
  more likely that hosting and delivering the models will be commoditized, much like how DO, Linode, Hertzer etc all commoditized VPSs and server hosting. And you'll end up paying for virtual hardware size (or compute resources) rather than tokens
• lenerdenator 2mo ago
  I wouldn't say "server under your desk", necessarily; more of an "Linux getting big" era of IT. If you want to host the model on the server under your desk, you can. If you want to build a data center on-prem to host it, you can. If you want to pay a cloud provider to host it at their data center until you figure out how to scale it without their help, you can. It's like when people were first building commercial services to support Linux-based OSes, and people were also still hacking on it on local machines. APIs may still have their place - maybe you just want to throw your devs a known quantity with all of the management built in - but it's not going to make Sam Altman a trillionaire, which is something anyone outside of the SV echo chamber could have figured out as soon as the first real competition to OpenAI emerged.