2 ms·
That is not a counterargument to cloud inference though. You can also run open weight models in the cloud, and it's still cheaper. So privacy really is the only
by int_19h 26d ago
That is not a counterargument to cloud inference though. You can also run open weight models in the cloud, and it's still cheaper. So privacy really is the only motivation to run on local hardware.
- anon373839 26d agoAh, no, that’s not cheaper. Renting GPUs adds up quickly and leaves you with nothing in the end. Renting tokens from open model providers is cheaper but it incurs the same issues: unexpected changes in model quality, inconsistent speeds, service outages.
- int_19h 23d agoRenting tokens is exactly what I had in mind, and I don't see how it has the same issues. Sure, if you go for absolute bottom of the barrel than you can expect "changes in model quality", but in general that is not so, and you can pin specific model releases etc. Inconsistent speeds and service outages apply to your own hardware as well (say, power goes down). This all isn't even unique to AI, it's generally why cloud compute quickly became dominant.