3 ms·
Local/open-weight models are already incredibly competent. Right now a Mac Studio with 256GB can be found for less than $5000, and an equivalent workstation wil
by rz2k 1y ago
Local/open-weight models are already incredibly competent. Right now a Mac Studio with 256GB can be found for less than $5000, and an equivalent workstation will likely be 50% cheaper in a year. If anything that price is higher because of the boom, rather than subsidized by a potential bubble. It can run a 8bit quant of GPT-OSS 120B, or 4bit quant of GLM-4.6 using only an extra 100-200W. That energy use comes out to about 100 Joules or 1/4 Wh per query and response, and is already competitive with the power efficiency of even Google's offerings.
I think that people doing work in many professions with these offline tools alone could more than double their productivity compared to their productivity two years ago. Furthermore if the usage was shared in order to lower idle time, such as 20 machines for 100 workers, the initial capital outlay is even lower.
Perhaps investors will not see the returns they expect, but it is difficult to image how even the current state of AI doesn't vastly change the economy. There could be significant business failures among cloud providers and attempts to rapidly increase the cost of admission to closed models, but there's essentially no possibility of productivity regressing to a pre-AI levels.
- tyleo 1y agoI have an M4 MBP and I also think Apple is set up quite nicely to take real advantage of local models. They already work on the most expensive Apple hardware. I expect that price to come down in the next few years. It’s really just the UX that’s bad but that’s solvable. Apple isn’t having to pay for each users power and use either. They sell hardware once and folks pay with their own electricity to run it.
- hiq 1y agoYour comment made me realize that there's also the benefit of not having to handle the hardware depreciation, it's pushed to the customer. And now Apple has renewed arguments to sell better machines more often ("you had ChatGPT 3-like performance locally last year, now you can get ChatGPT 4-like performance if you buy the new model"). I know folks who still use some old Apple laptops, maybe 5+ years old, since they don't see the point in changing (and indeed if you don't work in IT and don't play video games or other power-demanding jobs, I'm not sure it's worth it). Having new models with some performant local LLM built-in might change this for the average user.
- deleted 1y ago[deleted]
- fennecbutt 1y agoLmao it took Apple like a decade or soaafter everyone else to offer 16gb ram as default. You won't be getting cheap Apple machines chock full of ram any time soon, I can tell you that. That goes against Apple's entire pricing structure/money making machine.