4 ms·
Another possibility not really addressed here --- local LLMs. AI on hardware you own and control --- instead of a metered service provider. In other words, a
by jqpabc123 7mo ago
Another possibility not really addressed here --- local LLMs.
AI on hardware you own and control --- instead of a metered service provider. In other words, a repeat of the "personal computing" revolution but this time focused on AI.
TurboQuant could be a key step in this direction.
- netdevphoenix 7mo agoLocal LLMs don't sound profitable at all for those building them. If you really wanted a SOTA model, you would be paying eye watering amounts to own it unless you got an open sourced one.
- jqpabc123 7mo agounless you got an open sourced one. Ding, ding, ding --- we have a winner. https://techstartups.com/2026/03/26/nvidia-backed-ai-startup-reflection-eyes-2-5b-round-at-25b-valuation-to-challenge-deepseek-meta-and-mistral/ https://techstartups.com/2026/03/26/nvidia-backed-ai-startup... https://tiiny.ai/ https://tiiny.ai/
- schnitzelstoat 7mo agoYeah, I don't think local LLM's will keep up with what the massive corporations put out. But they might get to a level of performance where it just doesn't matter for most users. And people would prefer to run a model locally for 'free' (not counting the energy cost) rather than paying for an LLM subscription.
- zozbot234 7mo agoTurboQuant helps KV quantization which is not very relevant to local LLMs, since context size becomes most relevant when you run inference with large batches. For small-scale inference, weights dominate. (Even if you stream weights from SSD, you'll want to cache a sizeable fraction to get workable throughput, and that dominates your memory usage.)