3 ms·
You can fit a Q5 Llama-3-8b or many other models on new iPhones.
by a_wild_dandan 2y ago
You can fit a Q5 Llama-3-8b or many other models on new iPhones.
- ffgjgf1 2y agoIt’s not very good though and with 6 GB of memory iPhones would struggle anything else alongside it so forget multitasking or even running the full OS while it’s loaded.. Almost all Apple devices (including most Macs) have very low amounts of memory so Apple hadn’t really positioned themselves that well if their goal is running LLMs locally.
- d13 2y agoLlama 3 8b Q5 is within GPT4 territory. I’ve been using it daily, it’s astonishing.
- ffgjgf1 2y agoI found it to be not even remotely close outside of a few tasks/queries
- theshrike79 2y agoYou don't need a "phone AI" to solve general relativity. You need it to understand what you want to do within the limited scope of the phone. Basically I want Apple's AI to do what Shortcuts is capable of today, but instead of a janky Scratch programming style, I want it to respond to speech.