4 ms·
You will simply not get more value out of running a local model vs paying for a subscription/API from the cloud in 2026. There is no math that will make local m
by aurareturn 3mo ago
You will simply not get more value out of running a local model vs paying for a subscription/API from the cloud in 2026. There is no math that will make local models come out ahead in $/intelligence/token.*
The point of local models is privacy, offline use, and maybe no guard rails.
* Not talking about enterprises that buy DGX racks and host Chinese models for internal use.
- AureliusMA 3mo agoPoints are starting to be made in favor of value, to the contrary of what you are affirming. Specifically because the new open weights models lower the TCO of hardware in an environment where new open weights were previously thought to be a thing of the past.
- gilgoomesh 3mo ago[dead]
- Der_Einzige 3mo agoThere is for $/creativity/token. LLM sampling settings are poorly supported even in open source serverless providers but are the single best lever you have for getting better outputs in regards to creativity (and quality for long context or highly quantized models).
- aurareturn 3mo agoI'm pretty sure you can adjust the creativity for many Chinese model inference providers.
- Der_Einzige 3mo agoMost of them don't expose more than top_p/top_k/temperature. Those are woefully inadequate compared to what open source inference engines support.
- zargon 3mo ago> The point of local models is privacy, offline use, and maybe no guard rails. But also, control and consistency. A local model cannot be changed out under your feet like an API model can be.