4 ms·
I hope price increases will mean we see serious moves towards open source and/or on device models. Feels like that's where we should all be headed but the curre
by afavour 5mo ago
I hope price increases will mean we see serious moves towards open source and/or on device models. Feels like that's where we should all be headed but the current subsidy is a distraction.
- Fire-Dragon-DoL 5mo agoI agree but with the current hardware pricing it is a big problem
- nradov 5mo agoJust be patient for a few years. Even with the end of Moore's Law, compute hardware pricing always declines over the long run.
- Fire-Dragon-DoL 5mo agoYeah I am waiting but: If anything breaks on my pc, it's going to be really rough. My kids grow fast,a few years is a lot for them too. And i have a bunch of games I want to play with my wife that require 2 computers sadly
- miroljub 5mo agoThere is no subsidy in interference. Serving existing models is a highly profitable business. The proofs are many independent companies serving models for profit. What is expensive is model development and training, but again, that's nothing inherent to technology. It's just that Anthropic and other western vendors made a business decision to use cheap investors money to brute force a new model development and market grab instead of investing in cost optimizations for model training and inference like Chinese model developers and providers.
- eigencoder 5mo agoAs I understand it, those cost optimizations relied on someone else brute forcing new model development. If everyone was trying to do it cheaper, we wouldn't have the models we have today.
- zaphar 5mo agoMany of the chinese developers are just drafting off of Anthropic, OpenAI, and Gemini though. Distillation is leveraging those models to achieve their leaps in training. It's not obvious to me that Anthropic and do the same. Someone has to build the advanced model to draft off of.
- deleted 5mo ago[deleted]
- HWR_14 5mo agoI would self-host if it didn't cost so much for the hardware. But right now that's cost prohibitive for most individuals and small or medium businesses.
- throwaway132448 5mo agoFunny how that happened, isn’t it?
- nonethewiser 5mo agoHow what happened? How everyone stopped having cutting edge graphics cards in all their devices? Most people have never been in a position to be able to do local inference on cutting edge models.
- whizzter 5mo agoMaybe that's the reason why AI firms tried to monopolize ram, chips like AMD AI MAX 395 with AMD not being beholden or useful to the AI firms coupled with 128gb+ ram would probably have been capable of making home-hosted open models competitive enough that the AI firms valuations would've sunk like stone before their IPO's. The question is if the MoE advances and compression (TurboQuant) makes common computers fast enough to run local models that are "Good Enough".
- 999900000999 5mo agoOn device is still meh. Realistically Apple needs to put a few teams on it, and have built in support for running LLMs on OSX. I assume if you integrate this on an OS level you might be able to pull off some additional tricks. So far most of the tools for running LLMs locally have been driven by volunteers or very small startups. Microsoft could also do this, but they're too busy trying to upsell everyone. Microsoft wants to sell you on paying for each token.
- CharlieDigital 5mo agoApple has the edge here since they fully control the hardware and can baseline their implementation. It feels like Microsoft's task is much harder because there's a much broader range of hardware and capabilities that they need to support.
- 999900000999 5mo agoMicrosoft sells hardware. Surface laptops. Microsoft has spent a lot of time telling people to buy AI ready PCs. I just don't think it's in their business model to offer on device LLM. They want you to subscribe to Copilot. Which encapsulates the entire enshitication of Windows. Microsoft demands more money. Subscribe to gamepass, subscribe to OneDrive!
- CharlieDigital 5mo agoI'm aware of the Surface line, but 1) they don't own the hardware stack nearly to the same degree as Apple does, 2) it makes up only a small slice of the ecosystem and is a more premium product as far as PCs go. Windows ecosystem has far more variance in terms of CPU + GPU capabilities. I can see Microsoft pushing the subscription angle, but I do think there is a place for on-device capabilities since those will always be more responsive for small tasks.
- 999900000999 5mo agoI agree with your points. While I'm just thinking out loud, I can imagine either anthro or open AI releasing hardware for on-prem solutions in the near future. I believe a few small companies have already specialized in this. But imagine a situation where maybe you pay a small subscription and Open AI allows you to download non open source models guaranteed to work to a degree. It's going to be an interesting decade.