4 ms·
> within a few years we will be running local models as good as today’s frontier Unless there isn't some important breakthrough in hw production or in models a
by vb-8448 5mo ago
> within a few years we will be running local models as good as today’s frontier
Unless there isn't some important breakthrough in hw production or in models architecture, it's quite the opposite: bigger, more expensive and more energy-intensive hw is needed today compared to 1 or 2 years ago.
- ls612 5mo agoAs good as today’s frontier. Gemma 4 today is roughly equivalent to the frontier a year and a half ago at gpt 4o tier.
- antisthenes 5mo agoWhat's the cheapest PC you can buy today that will comfortably run Gemma 4 and everything else you want it to run at the same time? And how many tokens would that buy?
- ls612 5mo agoI run it on my 4 year old MBP and get 10 tok/s. With the RAM shortage buying anything new today is a nightmare but anyone with a reasonably modern Mac could run it at q6 probably. It is mostly a toy as 4o models weren’t really suitable for real work IMO but at least it won’t ever give me a refusal.
- jazzyjackson 5mo agoAt 10toks, are you using it interactively or do you submit a prompt and come back to it later? I always thought it would make sense to just do conversations over email, asynchronously, the model can take all the time it needs and get back to me when it has an answer.
- ls612 5mo ago10 tok/s is around the borderline of interactive being good. I did the math and it is mostly bottlenecked by memory bandwidth, so in the future I can expect to run a similarly sized model on my 4090 once it gets retired from gaming service and get ~25 tok/s which will be very usable.
- evgen 5mo agoI can run qwen3.6-27b on a four year-old Macbook Pro that dominates ChatGPT-4o (the frontier model from 2 years ago) and is competetitve against early ChatGPT-5 versions. We are also getting a lot smarter about using and deploying these local models. Your entire AI stack from two years ago would be absolutely crushed by a todays local LLM models and a high-end local inference system when combined with a good modern coding agent.
- vb-8448 5mo agoToday open weights frontier models cannot run locally, unless quantization is used. Deep seek v4 pro require almost 1 TB of RAM in INT4. I hardly doubt there will be consumer grade HW to run it in 2 years either. And deep seek v4 pro is not even close to OAI or anthropic frontier models.
- chermi 5mo agoPer frontier token. You're not calculating the cost of a fixed quality asset here. Old hw running non-frontier models will be very valuable. In fact, we have two direct examples: older server gpus actually appreciating and the very obvious fact that not everyone always use MAX FULL EFFORT BEST MODEL no matter what.
- deleted 5mo ago[deleted]
- vb-8448 5mo agoNot consumer hw ... and if we speak about local llm we cannot assume most of us can put a rack in their basement Already today is not possible to run deep seek v4 pro locally, and I cannot imagine that in 2 years we will be.