3 ms·
Run an incredible 400B parameters on a handheld device. 0.6 t/s, wait 30 seconds to see what these billions of calculations get us: "That is a profound observ
by causal 7mo ago
Run an incredible 400B parameters on a handheld device.
0.6 t/s, wait 30 seconds to see what these billions of calculations get us:
"That is a profound observation, and you are absolutely right ..."
- WarmWash 7mo agoI don't think we are ever going to win this. The general population loves being glazed way too much.
- baal80spam 7mo ago> The general population loves being glazed way too much. This is 100% correct!
- tombert 7mo agoThat's an astute point, and you're right to point it out.
- actusual 7mo agoYou are thinking about this exactly the right way.
- 9dev 7mo agoYou’re absolutely right!
- otikik 7mo agoThe other day, I got: "You are absolutely right to be confused" That was the closest AI has been to calling me "dumb meatbag".
- keybored 7mo agoPoor “we”. “They” love looking at their own reflection too much.
- intrasight 7mo agoBetter than waiting 7.5 million years to have a tell you the answer is 42.
- thinkingtoilet 7mo agoMaybe you should have asked a better question. :P
- deleted 7mo ago[deleted]
- whyenot 7mo agoShould have used a better platform. So long and thanks for all the fish.
- ep103 7mo agoSome one should let Douglas Adams know the calculation could have been so much faster if the machine just lied.
- Aurornis 7mo agoI thought you were being sarcastic until I watched the video and saw those words slowly appear. Emphasis on slowly.
- amelius 7mo agoI mean size says nothing, you could do it on a Pi Zero with sufficient storage attached. So this post is like saying that yes an iPhone is Turing complete. Or at least not locked down so far that you're unable to do it.
- zozbot234 7mo agoYou need fast storage to make it worthwhile. PCIe x4 5.0 is a reasonable minimum. Or multiple PCIe x4 4.0 accessed in parallel, but this is challenging since the individual expert-layers are usually small. Intel Optane drives are worth experimenting with for the latter (they are stuck on PCIe 4.0) purely for their good random-read properties (quite aside from their wearout resistance, which opens up use for KV-cache and even activations).
- r_lee 7mo agoI too thought you were joking laughed when it slowly began to type that out
- vntok 7mo ago2 years ago, LLMs failed at answering coherently. Last year, they failed at answering fast on optimized servers. Now, they're failing at answering fast on underpowered handheld devices... I can't wait to see what they'll be failing to do next year.
- ezst 7mo agoProbably the one elephant in the roomy thing that matters: failing to say they don't know/can't answer
- BirAdam 7mo agoThe speed on a constrained device isn't entirely the point. Two years ago, LLMs failed at answering coherently. Now... You're absolutely right. Now, LLMs are too slow to be useful on handheld devices, and the future of LLMs is brighter than ever. LLMs can be useful, but quite often the responses are about as painful as LinkedIn posts. Will they get better? Maybe. Will they get worse? Maybe.
- vntok 7mo ago> Will they get better? Maybe. Will they get worse? Maybe. I find it hard to understand your uncertainty; how could they not keep getting even better when we've been seeing qualitative improvements literally every second week for months on end? These improvements being eminently public and applied across multiple relevant dimensions: raw inference speed (https://github.com/ggml-org/llama.cpp/releases https://github.com/ggml-org/llama.cpp/releases), external-facing capabilities (https://github.com/open-webui/open-webui/releases https://github.com/open-webui/open-webui/releases) and performance against established benchmarks (https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks)
- mememememememo 7mo agoMake "you are absolutely right" it's own token?