4 ms·
> Want it to go away, almost like magic? Local inference. When its under your control, and no longer being forced to hold it wrong, all of the common LLM defect
by Aurornis 3mo ago
> Want it to go away, almost like magic? Local inference. When its under your control, and no longer being forced to hold it wrong, all of the common LLM defects will go away.
This is just not true. Any local LLM you can host on consumer-accessible hardware has all of these defects, too. Adjusting the knobs doesn’t solve everything.
The closest you can get to frontier performance is Kimi K3, but you’re not hosting that unless your budget is on the order of a nice house in a good metro area.
I like my local LLMs as much as the next person and my office is currently uncomfortably warm from the amount of compute happening, but I would never agree that local LLMs solve all of the common LLM defects. This is peak wishful thinking.
In my experience, the local models and even the larger ones that we can’t run at home suffer more from long context degradation than the frontier models. You are exactly right that you need to manage context length, but even at fp16/bf16 the local models have a lower ceiling for usable context length in my experience.
- slopinthebag 3mo agoDo you work for a frontier lab by any chance?
- Aurornis 3mo agoNo.
- dingnuts 3mo ago[dead]
- nzeid 3mo agoThis is a bit of a strawman. The parent comment wasn't suggesting that the inherent defects magically disappear nor was it suggesting that local inference today is sufficient for all tasks. At the heart of it, self-hosting liberates your use cases from all the horribly opaque configuration, shadow prompting, etc. And local models are only getting better and more diverse every month.
- joshmoody24 3mo ago> Want it to go away, almost like magic? True, almost like magic and magic are not the same thing, but I'd hesitate to call this a strawman.
- john_strinlai 3mo ago>The parent comment wasn't suggesting that the inherent defects magically disappear that is almost exactly what they said though..? "Want it to go away, almost like magic? Local inference. [...] all of the common LLM defects will go away.
- pessimizer 3mo ago"[...]" ----- edit: I suppose I have to spell this out. "[...]" = "When its under your control, and [you're] no longer being forced to hold it wrong," ≈ "liberates your use cases from all the horribly opaque configuration, shadow prompting, etc." and "Almost like magic" is cliché hyperbole. It also means "not magic" in the same way that "almost like a dog" means "not a dog." It was immediately followed by the explicit clarification that was surgically omitted above, in absolute and conscious bad faith. Hence, "[...]"
- john_strinlai 3mo agoi omitted it because it doesnt change my point at all and it makes the quoted text smaller. not because of "absolute and conscious bad faith". people can just read the original from the comment. its not like im quoting from a different website and trying to hide something. look 2 comments above mine and it is right there in full. anyone reading my comment would have seen the original comment already... i have no idea why you took such offense to my comment, but you're reading way too much into it.
- throw10920 3mo ago> "Almost like magic" is cliché hyperbole. It also means "not magic" in the same way that "almost like a dog" means "not a dog." It was immediately followed by the explicit clarification that was surgically omitted above, in absolute and conscious bad faith. Incorrect. The omitted text doesn't change the relevant meaning of the quotation in the slightest. Why are you taking such offense? > Want it to go away, almost like magic? Local inference. When its under your control, and no longer being forced to hold it wrong, all of the common LLM defects will go away. > "Want it to go away, almost like magic? Local inference. [...] all of the common LLM defects will go away. There's no difference between these two statements in the relevant context of the article, which is specifically "long policy documents do not reliably govern agents", or the comments above, which are specifically addressing how that issue is not solved by local agents.
- KerrAvon 3mo agoWe're on the cusp of Kimi K3 becoming usable on sub-10k hardware. https://github.com/gavamedia/deltafin https://github.com/gavamedia/deltafin
- edot 3mo ago14 seconds per token? Not tokens per second. Seconds per token? That’s nowhere near the cusp!
- dotancohen 3mo agoIt's need it to be an order of magnitude cheaper, but for some queries ("What to discuss at tomorrow's meeting") I can wait 12+ hours.
- throw10920 3mo agoI'd rather just use my actual human brain to compute the answer at that point. I don't see the value at throughput that is this low.
- dotancohen 3mo agoI'd rather not be thinking about tomorrow's meeting all night! Just trust "the robot has got this"... and scramble in the morning when the robot has failed. But at least I got a good night's sleep!
- antonvs 3mo agoAt that rate, it would take me a mere 15 days to generate the number of tokens I typically use in a day. "Yes, I'm using AI to speed up development. I'll submit that PR in two weeks time!"
- adrian_b 3mo agoThat speed is significantly lower that what can be attained with streaming from SSDs. Not only in big desktops, but even in most recent mini-PCs, it is possible to read concurrently from one PCIe 5.0 SSDs and one PCIe 4.0 SSD, at a total sustained reading throughput of around 20 GB/s. With an optimized inference implementation, it should be possible to overlap completely the computations with streaming weights from SSDs. This should improve the inference speed to around at least 1 token per second, on a cheap computer, under $2000 even at the current super-inflated prices. There are enough tasks where this would be useful. Obviously one should use for most tasks a fast small LLM and use the big one only when this actually saves time.