3 ms·
I would self-host if it didn't cost so much for the hardware. But right now that's cost prohibitive for most individuals and small or medium businesses.
by HWR_14 5mo ago
I would self-host if it didn't cost so much for the hardware. But right now that's cost prohibitive for most individuals and small or medium businesses.
- throwaway132448 5mo agoFunny how that happened, isn’t it?
- nonethewiser 5mo agoHow what happened? How everyone stopped having cutting edge graphics cards in all their devices? Most people have never been in a position to be able to do local inference on cutting edge models.
- whizzter 5mo agoMaybe that's the reason why AI firms tried to monopolize ram, chips like AMD AI MAX 395 with AMD not being beholden or useful to the AI firms coupled with 128gb+ ram would probably have been capable of making home-hosted open models competitive enough that the AI firms valuations would've sunk like stone before their IPO's. The question is if the MoE advances and compression (TurboQuant) makes common computers fast enough to run local models that are "Good Enough".