3 ms·
I would say that is highly unlikely if by SOTA models you are not just referring to coding benchmarks but more general purpose ability and domain-specific knowl
by ghrl 4mo ago
I would say that is highly unlikely if by SOTA models you are not just referring to coding benchmarks but more general purpose ability and domain-specific knowledge. For example Kimi 2.6, which is comparable to Opus 4.6, is roughly 500+GB large, and I don't see how that would run on consumer hardware anytime soon.
Besides, this is not just about the technical feasibility, but also economically not viable whatsoever. Why should consumer laptops be capable of running such models, when they would be massively underutilized most of the time, when inference providers can produce the same results faster, cheaper and a lot more viable economically?
- henry2023 4mo agoBecause privacy has perceived value.
- sourcecodeplz 4mo agoIt runs right now on 512gb RAM Macs and PCs.
- Our_Benefactors 4mo agoIt runs like shit though in terms of tokens/second and still has a reduced context window. Vs a single claude prompt can easily get into 300k tokens without breaking a sweat. I want local AI to be a thing but the hardware isn’t here yet, because the only options are a Mac Studio or DGX machines strapped together. RAM prices needs to crash before local AI has a chance at actually competing.
- zozbot234 4mo agoThe more recent Chinese models are no longer heavily limited by context size. It can easily fit in RAM on a prosumer laptop. (You can also use swap space to extemd that, since context is only written to once per inference, thus a relatively mild wear-and-tear concern.)
- Our_Benefactors 4mo agoClaude has 1M context window for the enterprise. 128k feels like a toy in comparison.
- sourcecodeplz 4mo agoDeepseek pro/flash both have 1m.
- ATMLOTTOBEER 4mo agoYou’re right, and it feels like these people saying otherwise either don’t use these tools professionally (and therefore can’t tell a difference between local/cloud models) or literally just haven’t tried running local models As soon as I can buy hardware for less than 5k that runs an opus 4.6+/5.5 model locally I will do it instantly