7 ms·
A quick survey of the thread seems to indicate the 7b parameter LLaMA model does about 20 tokens per second (~4 words per second) on a base model M1 Pro, by tak
by fwlr 4y ago
A quick survey of the thread seems to indicate the 7b parameter LLaMA model does about 20 tokens per second (~4 words per second) on a base model M1 Pro, by taking advantage of Apple Silicon’s Neural Engine.
Note that the latest model iPhones ship with a Neural Engine of similar performance to latest model M-series MacBooks (both iPhone 14 Pro and M1 MacBook Pro claim 15.8 teraflops on their respective neural engines; it might be the same exact component in each chip). All iPhone 14 models sport 6GB integrated RAM; the MacBook starts at 8GB. All of the specs indicate an iPhone 14 Pro could achieve similar throughput to an M1 MacBook Pro.
Some people have already had success porting Whisper to the Neural Engine, and as of 14 hours ago GGerganov (the guy who made this port of LLaMA to the Neural Engine and who made the port of Whisper to C++) posted a GitHub comment indicating he will be working on that in the next few weeks.
So. With Whisper and LLaMA on the Neural Engine both showing better than real-time performance, and Apple’s own pre-existing Siri Neural TTS, it looks like we have all the pieces needed to make a ChatGPT-level assistant operate entirely through voice and run entirely on your phone. This is absolutely extraordinary stuff!
- schappim 4y ago> Some people have already had success porting Whisper to the Neural Engine, and as of 14 hours ago GGerganov (the guy who made this port of LLaMA to the Neural Engine and who made the port of Whisper to C++) posted a GitHub comment indicating he will be working on that in the next few weeks. He has already done great work here: https://github.com/ggerganov/whisper.cpp https://github.com/ggerganov/whisper.cpp
- sho_hn 4y agoThe 7b model specifically is not quite "ChatGPT-level" though, is it?
- doctoboggan 4y agoAccording to Meta's benchmarking[0] it is comparable on many metrics. I haven't used it myself so I can't say for sure if that is the case when actually using it. [0]: https://arxiv.org/pdf/2302.13971.pdf https://arxiv.org/pdf/2302.13971.pdf
- redox99 4y agoThat's GPT3, not ChatGPT.
- Kelamir 4y agoI don't understand this topic well, but given premise that GPT3 and ChatGPT are different only that ChatGPT includes RLHF(Reinforcement Learning from Human Feedback), and LLaMA 7b is comparable to GPT3 on a number of metrics, it would follow that if we were to improve LLaMA 7b with RLHF, the 7b model would be similar to ChatGPT. Is that correct?
- throwaway1851 4y agoI’m interested in this as well. Comparatively little attention has been paid to those 7B model results, but they look quite good against 175B GPT-3. As for ChatGPT, that is GPT-3.5 (same 175B model, but with instruction fine-tuning), plus the RLHF.
- DavidSJ 4y agoGPT 3.5 likely differs from the original GPT 3 by more than instruction fine-tuning. For example, it was probably retrained under Chinchilla scaling laws [1], with a lot more data and maybe a somewhat smaller parameter count. There are many variants of GPT-3 and GPT-3.5, and based on the performance numbers in Meta’s paper, it looks like they’re comparing against the very first version of GPT-3 from 2020. [2] [1] https://arxiv.org/abs/2203.15556 https://arxiv.org/abs/2203.15556 [2] https://arxiv.org/abs/2005.14165 https://arxiv.org/abs/2005.14165
- popinman322 4y agoYou're likely right that applying RLHF (+ fine-tuning with instructions) to LLaMA 7b would produce results similar to ChatGPT, but I think you're implying that that would be feasible today. RLHF requires a large amount of human feedback data and IIRC there's no open data set for that right now.
- toxik 4y agoNone of the Meta models are RLHF tuned, as far as I know.
- RonnieOwnsLexus 4y agoBut wont it be that in real life no one would want to run a voice command which consumes lot of CPU and battery as opposed to making a network call to a service which has this model hosted ? Agreed that this can always be improved and hardware can get more efficient and better to but at the end of the day, would it ever be better then an API call ?
- Tepix 4y agoPrivacy concerns are justified. It's not just that, this can also work completely offline.
- sebzim4500 4y agoUltimately, no amount of technology will ever beat the speed of light. Running locally will always have a lower latency floor.
- brookst 4y agoTheoretically yes. But in the real world, no. Simple thought experiment: you want to know how many tons of copper are mined in the US each year. Lowest possible latency is calculating this in your head, most likely using data you don’t have. Looking it up online is a lot, lot faster. In some far future world maybe every transistor will include the sum total of human knowledge up to the nanosecond, but that’s a pretty far future. There are many things where running locally means a higher latency floor.
- Someone 4y ago> All of the specs indicate an iPhone 14 Pro could achieve similar throughput to an M1 MacBook Pro. Battery capacity and thermals are different and might be problematic. The phone might throttle performance earlier. > it looks like we have all the pieces needed to make a ChatGPT-level assistant operate entirely through voice and run entirely on your phone. As a demo, yes, but would loading the model be fast enough for Siri-like responsiveness? You also would want to run other programs alongside it. And of course, for Apple to adopt something like this, we would have to get rid of the tendency of these models to derail conversations. Put in something somewhat sexist/racist/…, and it will reply with something a bit more sexist/racist/…) But yes, it would be a cool demo.
- j45 4y agoSiri doesn’t seem as fast or responsive compared to Google assistant at times.
- sgt 4y agoSiri is sometimes busy doing laundry or Gods know what. I think the quality of Siri is much better than Google Assistant but I wonder about the lag.
- j45 4y agoReally? I find Siri can’t understand anything slightly more than basic instructions. Google assistant can seem to do more
- andsoitis 4y agoI'm very interested in this space. Can you share an example that illustrates the difference in "understanding" between the two?
- cromka 4y agoJust recently Siri would belly-up on “Turn off Living Room lightS” — it would only work if I said “light” (singular). Extremely frustrating. They fixed it, I think, but this arbitrariness and many other make me think Siri is more quirk- and algorithms-based than a true AI.
- sebzim4500 4y ago>20 tokens per second (~4 words per second) How can there be 5 tokens per word, when they have more than half the vocabulary as GPT-2/3 which has 1.3 tokens per word? I would have guessed more like 1.5 tokens per word.
- fwlr 4y agoOh, it’s probably higher than four words per second, then. I assumed tokens was characters and used the standard “there are five characters in a word” rule of thumb.
- MacsHeadroom 4y agoIt's about 4 charcters per token. So just over 1 token per word. I just round to 1 token per word since text most people generate does not use larger words and because larger common words are still encoded as one token (e.g. HackerNews is probably one token despite being 10 characters).
- Taek 4y agoI typically see people claim 2-3 tokens per word.
- snickmy 4y agoI wish we could start having open source TTS models with similar performance. So far Tortoise TTS is not there yet. Im not sure if Siri neural TTS is offered for 3rd party apps.
- gitfan86 4y agoI would also expect 10x improvements over the next year due to optimizations found throughout the stack.
- return_to_monke 4y agoi think voice assistants can perform actions on phones (eg "open app, message Alice, call Bob, turn off Bluetooth"). This couldn't do that (I think), which is an obvious drawback
- amusedcyclist 4y ago4 words a second doesn't seem fast enough for a voice assistant ?
- schappim 4y agoI've had difficulty obtaining useful results from the smaller (7B-sized) models. The issue lies in the content, not the speed. If you could stream the text-to-speech, the speed alone would be satisfactory.
- amusedcyclist 4y agoYou're right I overestimated how fast we talk!
- fwlr 4y agoSome rules of thumb I use for estimating this kind of stuff 100wpm: Max typing speed 200wpm: Max speaking speed 300wpm: Max listening speed, max reading speed with subvocalisation 900wpm: Max reading speed without subvocalisation
- mromanuk 4y agoDoin napkin math, this model should be hitting 900wpm
- mromanuk 4y agoIt's faster than that [0], 20 token/s, should be approximately 15 words per second. 0: https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them https://help.openai.com/en/articles/4936856-what-are-tokens-...
- DennisAleynikov 4y agoIf I may, this library runs LLaMA on CPU. There is no way to run it on the Neural Engine yet. The optimization in this case only seems to refer to the 4bit model loading method (to be friendlier to the arm64 CPU) GeoHot has tinygrad running LLaMA on Metal (but only the 7B model) that's the closest I've seen to taking advantage of apple silicon. Neural Engine implementation would be awesome
- fwlr 4y agoOh shit, I took a closer look and you’re right. The repo was also helpfully updated with a note to this effect: “The Accelerate framework is actually currently unused since I found that for tensor shapes typical for the Decoder, there is no benefit compared to the ARM_NEON intrinsics implementation. Of course, it's possible that I simlpy don't know how to utilize it properly. But in any case, you can even disable it with LLAMA_NO_ACCELERATE=1 make and the performance will be the same, since no BLAS calls are invoked by the current implementation”. No Joi in my pocket just yet :( Because of this I re-checked my claims about the Whisper speed up from the Neural Engine and that does look legit, 6x at least. So the Neural Engine does have the chops for this workload, it just isn’t being used in this repo. It may not be LLaMA, but I sure hope someone gets an LLM running on the ANE sooner rather than later.
- ggerganov 4y agoOur investigations indicate that it might not be possible to achieve ANE performance improvement over CPU for LLM Decoder inference with batch size of 1 [0]. Just to make it clear - I'm no expert in Core ML / ANE, so these conclusions could be totally wrong. [0] https://github.com/ggerganov/whisper.cpp/discussions/548#discussioncomment-5265849 https://github.com/ggerganov/whisper.cpp/discussions/548#dis...
- fwlr 4y agoDon’t sell yourself short! (And you have my apologies in advance if my excited comment above has created any extra work for you)
- ar9av 4y agothe potential drawbacks of relying entirely on voice-operated assistants like ChatGPT. There are concerns around privacy and the use of personal data, as well as the potential for bias and inaccuracies in the responses generated by these models. It's important to strike a balance between the convenience and benefits of these technologies and the potential risks and limitations they bring. Nonetheless, the advancements being made in this field are impressive and it will be interesting to see how they develop in the future.
- Centigonal 4y agoThat's very ChatGPT of you to say!