7 ms·
LLaMa running at 5 tokens/second on a Pixel 6
- superkuh 4y agoUntil it thermally throttles 40 seconds later. But yeah, it's really cool how many platforms the vanilla code in llama.cpp can be easily compiled on. And somehow I doubt they did the quantization step on the Pixel itself. My favorite was the person who did it on the rpi4. I know a guy working on getting it going on rpi3 but the ARM7/8 mixing , NEON support, and 64 bit ARM intrinsics are apparently non-trivial to convert.
- sebzim4500 4y ago>And somehow I doubt they did the quantization step on the Pixel itself You're probably right (because why would they?) but I don't see any reason they couldn't have done this if they wanted to.
- aquajet 4y agoI would have tried it but I didn't have enough storage on my phone to hold both the original and quantized weights.
- aquajet 4y agoI'm the original tweet author. Currently typing this from my Pixel after running it countless times :)
- mdrzn 4y agoI would need a step by step guide, but I would love to test it on my Galaxy S21 Ultra 5G. It has 16gb ram and I have about 350 GB available.
- naillo 4y agoThis is really cool but the output is such garbage at that weight size that you might as well be running a markov chain.
- lostmsu 4y agoFrom the video output seems fine. But if it is a trimmed version, it is wong to call it LLaMa.
- refulgentis 4y agoIt's nonsensical, celeb announces they're going to rehab and notes it (?) is an issue affecting all women, at least, earlier today (??), they also noted it wasn't drugs or alcohol this time, but, a life (???)
- ddren 4y agoWhat are you comparing it to? Without instruction tuning and a two character prompt "He" I am not sure why you would expect it to perform any better.
- refulgentis 4y agoI was replying to a comment that said it “seems fine.” It does not seem fine. It is incomprehensible and doesn’t match the results I’ve seen from 7B through 65B. It is true that RLHF could improve it, and perhaps then this severe of optimization will seem fine.
- tbalsam 4y agoI've heard a number of people say (from earlier) that the quantization and default sampling parameters is way wacked. Honestly even running that model size alone is the big achievement here and getting the accuracy to actually reach the benchmark is the beeg next step nao, I believe. <3 :'))))
- lostmsu 4y agoIf you run a quantized 60G model and the output is worse than raw 7G model, you can throw your quantizer out.
- Havoc 4y agoAny more details? I'm guessing they're leveraging the NPU in the pixel?
- saidinesh5 4y agoI think they are using llama.cpp without any NPU/TPU patches. By default it only runs on CPU with support for various SIMD extensions. https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp
- zodester 4y agoIt uses the ARM NEON extensions to the instruction set for SIMD (as far as I understand).
- __mharrison__ 4y agoI'm waiting until it runs on my C64...
- deleted 4y ago[deleted]
- tosh 4y agoDid anyone get this to run on an iPhone or in a browser yet?
- pzo 4y agoMost iphone have only 4GB RAM (and even latest iphone 14 has only 6GB RAM). Pixel 6 has 8GB RAM. But bigger issue is on iOS still OS limits how much RAM your app can use and might kill your app.
- londons_explore 4y agoI'm still amazed that Apple invests so much into every other bit of hardware on a high end phone, yet always gives you the bare minimum amount of RAM they can get away with. There are so many use cases (like this) that require more RAM. And even if a use case doesn't theoretically require more RAM, getting a developer to dedicate time to optimizing RAM is time taken away from making a wonderful app.
- squarefoot 4y ago> I'm still amazed that Apple invests so much into every other bit of hardware on a high end phone, yet always gives you the bare minimum amount of RAM they can get away with. Advanced hardware makes bullet points on advertising to sell the device; giving the bare minimum of RAM accelerates the device planned obsolescence, so that user will be forced to upgrade sooner to the next model.
- szundi 4y agoHow is that an iPhone 7 is completely current vs give me a branded Android from the same year of release that has even security updates, not even features.
- alden5 4y agoThe problem with iPhones is once updates stop there's nothing you can do. The iPhone 7 isn't current, it's stuck on iOS 15 while the newest is 16. And while the pixel 2 (which is only a month younger than the iPhone 7) only got official support up to Android 11; you actually own the device and can easily unlock the boot-loader to upgrade to Android 13.
- syntaxing 4y agoDoes this in theory mean it should be relatively easy to port to coral TPU?
- saidinesh5 4y agoAll their tensor/math magic seems to happen in https://github.com/ggerganov/llama.cpp/blob/master/ggml.h https://github.com/ggerganov/llama.cpp/blob/master/ggml.h . So maybe if you implement the ggml.c with tensorflow/libcoral - you'd have a chance.
- sottol 4y agoAfaik that TPU has only 8MB of RAM to fit models, you'd have to continuously stream the weights - can't imagine that's workable.
- OscarCunningham 4y agoThis would be useful for predictive text. That's exactly what LLMs are actually built for.
- a-dub 4y agoLSTMs have been in the Google keyboard for years...
- nshm 4y agoIt is not really llama, it is llama quantized to 4bit. Not even the quality of original 7B. I could also quantize it to 1 bit and claim it runs on my RPI3.
- mrWiz 4y agoThe 4 bit quantization performs well, though. Does your 1 bit version?
- nshm 4y agoDo you have the numbers? I suspect is is way worse. Original llama.cpp authors never measure any numbers as well.
- summarity 4y agoSome numbers here: https://github.com/qwopqwop200/GPTQ-for-LLaMa#result https://github.com/qwopqwop200/GPTQ-for-LLaMa#result
- sottol 4y agoThey're using GTPQ -- here you go: https://arxiv.org/abs/2210.17323 https://arxiv.org/abs/2210.17323 . The authors benchmarked two families of models over a wide range of numbers of params.
- ddren 4y agollama.cpp is using RTN at the moment.
- ddren 4y agoThe python implementation[1] ran some tests using the same quantization algorithm as llama.cpp (4 bit RTN). 1: https://github.com/qwopqwop200/GPTQ-for-LLaMa https://github.com/qwopqwop200/GPTQ-for-LLaMa
- 4y ago
- beiller 4y agoHere is a thread to tweak the parameters which the model seems very sensitive to: https://github.com/ggerganov/llama.cpp/issues/129 https://github.com/ggerganov/llama.cpp/issues/129
- nico 4y agoCould the model itself be used to tweak it’s own parameters iteratively?
- hackernewds 4y agoThat's Model Extraction basically
- nico 4y agoDo you have a link to an explainer of how it would work or a potential implementation?
- a-dub 4y agowould be even cooler if it employed the accelerator! (unless this ggml library is doing that under the hood) i assume it has unified memory, but maybe not little numbers...
- snapplebobapple 4y agoSo is this finally peak hipster coder and from this point on rust will diminish because all the cool kids start switching to zag?