6 ms·
Llama3 running locally on iPhone 15 Pro
- sciencesama 3y agonew apple pro ai model with 32gb ram !!
- deleted 3y ago[deleted]
- yaszko 3y agoApp live on App Store: https://apps.apple.com/us/app/local-brain/id6476066730 https://apps.apple.com/us/app/local-brain/id6476066730
- PcChip 3y agoWonder why it has a 2 star rating
- werber 3y agoOn the most recent iPhone pro I have a query running (about ~15 minutes so far) and the results are really good, just really slow, but I imagine the performance is worse on an older device
- kachapopopow 3y ago[dead]
- ValleZ 3y agoYou can run it on Android too. Not very fast though.
- ilteris 3y agoLink?
- radeeyate 3y agohttps://llm.mlc.ai/ https://llm.mlc.ai/
- freedomben 3y agoMore direct: https://llm.mlc.ai/docs/deploy/android.html https://llm.mlc.ai/docs/deploy/android.html APK download link: https://github.com/mlc-ai/binary-mlc-llm-libs/releases/download/Android/mlc-chat.apk https://github.com/mlc-ai/binary-mlc-llm-libs/releases/downl...
- ValleZ 3y agoI wonder why not to publish it on Google Play?
- ValleZ 3y agohttps://play.google.com/store/apps/details?id=us.valkon.privateai&hl=en_US https://play.google.com/store/apps/details?id=us.valkon.priv...
- noiv 3y agoFor 3 seconds I was hoping it might support 13 mini, but 4GB of RAM is not enough :(
- TMWNN 3y agoMy current and previous MacBooks have had 16GB and I've been fine with it, but given local models I think I'm going to have to go to whatever will be the maximum RAM available for the next one. It runs 13b models quite well with Ollama, but I tried `mixtral-8x7b` and saw 0.25 tokens/second speeds; I suppose I should be amazed that it ran at all. Similarly, I am for the first time going to care about how much RAM is in my next iPhone. My iPhone 13's 4GB is, as in your case, inadequate.
- gcanyon 3y agoI recently upgraded from my M1 Air specifically because I had purchased it with 8gb -- silly me. Now I have 24gb, and if the Air line had more available I would have sprung for 32, or even 64gb. But I'm not paying for a faster processor just to get more memory :-/
- regularfry 3y agoI got an 8GB M1 from work, and I've been frankly astonished with what even this machine can do. Yes, it'll run the 4bit llama3 quants - not especially fast, mind, but not unusably slow either. The problem is that you can't do a huge amount else.
- Havoc 3y agoNo luck on 14 pro max either
- PixelPioneerZZ 3y agoIt's fascinating. I hope to see more powerful large models on various mobile devices in the future.
- aussieguy1234 3y agoThis should be attempted with phi-3 when the weights are released tomorrow.
- eckelhesten 3y agoI’m sure my iPhone 14 Pro with 6GB RAM can handle the 4GB weights no? Says device is unsupported :(
- __rito__ 3y agoWhich app is this? Does anything similar exist for Android?
- dTal 3y agoOn Android you can simply run vanilla llama.cpp inside a terminal, or indeed any stack that you would run on a Linux desktop that doesn't involve a native GUI.
- iAkashPaul 3y agoYep, termux is a good way to do this. Llama.cpp has Android example as well, I forked it here GitHub.com/iakashpaul/portal you can try it with any supported GGUF/Q4+Q8 models
- iAkashPaul 3y agoYou can either modify the Android example inside llama.cpp or my fork of it at GitHub.com/iakashpaul/portal Increase the Ctx to more than 100 & link to any Q4 GGUF of 7B
- Fubarberry 3y agoThere's an app called Private AI that will let you run models locally on Android. It has a few smaller models available for free to try it out, but the larger models like Llama 3 (or the option to use your own downloaded models) require a $10 unlock purchase.
- SushiHippie 3y agoSeems like it is the 3bit quantized version? (Judging by the file size) And does anyone know how many tokens per second it can run?
- toxypol 3y agowhat is with the BS answer lol
- SushiHippie 3y agohttps://en.m.wikipedia.org/wiki/Fixed-point_combinator#Y_combinator https://en.m.wikipedia.org/wiki/Fixed-point_combinator#Y_com...
- TYPE_FASTER 3y agoAnybody else running into issues downloading models?
- yohannparis 3y agoThis is quite impressive to be honest. The chat is answering at a speed of one word per few/several seconds. But still, this a nice feat. Example recording for the curious: https://www.youtube.com/watch?v=nZEvUj-QTrI https://www.youtube.com/watch?v=nZEvUj-QTrI
- bezier-curve 3y agoOn my S24 Ultra, I am seeing it generate several words a second.
- sestep 3y agoI'm so spoiled by the quality of modern closed-source models, I laughed out loud when the beginning of its answer just said [duplicate].
- mariuz 3y agoRelated with GitHub link https://twitter.com/awnihannun/status/1782436898285527229 https://twitter.com/awnihannun/status/1782436898285527229
- aNoob7000 3y agoI wonder if Apple will bump up the amount of RAM in iPhones due to AI. It seems like most LLMs require a large amount of memory. They've been stingy on increasing RAM compared to Android phones.
- MPSimmons 3y agoI can't wait until Groq or someone else release tiny mobile inference engines specifically for phones and the like.
- Largeapplemodel 3y agoThere's already tiny LLMs for this. They're bad. Because it's not enough information to be coherent.
- MyFirstSass 3y agoIt's one of the most infuriating things about Apple. The ram tax so absurd it's bordering on criminal, but it also just seems stupid, because if they hadn't put 8gb's of ram in the new smallest macbook air m2 their whole lineup would be more than capable at running local quality LLM's, or double their gaming devices because of their awesome chipset giving them 16gb's of vram essentially, but no, not now when 25% have low ram, ie. no new OS LLM updates. Also we can't have gaming because half their new sold devices have shit ram, so they also kind of already ditched their "gaming" plan they just got started on a year ago - all because they wan't to push products with ram levels from 10 years ago - bizarre! They must be betting on local AI as a "pro" feature only.
- cqqxo4zV46cp 3y ago8GB for a premium device in 2024 is a hard ask, completely agree. But I hold absolutely zero hard feelings toward Apple for not catering to gamers as a demographic Most importantly, though, we are talking about iPhones here. I can’t say I’ve ever thought to myself “gosh, I wish my phone had more RAM!” in…over a decade?
- mariuz 3y agoRelated with GitHub link "Next level: QLoRA fine-tuning 4-bit Llama 3 8B on iPhone 15 pro. Incoming (Q)LoRA MLX Swift example by David Koski: https://github.com/ml-explore/mlx-swift-examples/pull/46 https://github.com/ml-explore/mlx-swift-examples/pull/46 works with lot's of models (Mistral, Gemma, Phi-2, etc)" https://twitter.com/awnihannun/status/1782436898285527229 https://twitter.com/awnihannun/status/1782436898285527229
- woadwarrior01 3y agoIs this news? I've got a nearly year old app that supports over 2 dozen local LLMs with support for using them with Siri and Shortcuts. I added support for Llama 3 8B the day after it came out and also Eric Hartford's new Llama 3 8B based Dolphin model. All models in it are quantized with OmniQuant. On iOS, 7B and 8B ones are 3-bit quantized and smaller models are 4-bit quantized. On the macOS version all models are 4-bit OmniQuant quantized. 3-bit Omniquant quantization is quite comparable in perplexity to 4-bit RTN quantization that all the llama.cpp based apps use. https://privatellm.app/ https://privatellm.app/ https://apps.apple.com/app/private-llm-local-ai-chatbot/id6448106860 https://apps.apple.com/app/private-llm-local-ai-chatbot/id64...
- davedx 3y agoNice. What is battery life like under heavy use? I was reading a thread on the llama.cpp repo earlier where they were discussing whether it was possible (or attractive) to add neural engine support in some form.
- woadwarrior01 3y agoWith bigger 7B and 8B models, the battery life goes from a over a day to a few hours on my iPhone 15 Pro. The 8B model nominally works on 6GB phones but it's quite slow on them. OTOH, it's very usable on iPhone 15 Pro/ Pro Max devices and even better on M1/M2 iPads. Every framework: llama.cpp, MLX, mlc-llm (which I use) all only use the GPU. Using the ANE and perhaps the undocumented AMX coprocessor for efficient decoder only transformer inference is still an open problem. I've made early some progress on quantised inference using ANE, but there 're still a lot of issues to be solved before it is even demo ready, let alone a shipping product.
- davedx 3y agoSuper interesting, thank you!
- Starlevel004 3y ago[flagged]
- deleted 3y ago[deleted]