4 ms·
If you haven't tried it yet, and you have a modern enough phone, I strongly recommend trying out Mistral 7B on your mobile device. I've been running it on an i
by simonw 3y ago
If you haven't tried it yet, and you have a modern enough phone, I strongly recommend trying out Mistral 7B on your mobile device.
I've been running it on an iPhone 15 using this app: https://llm.mlc.ai/#ios https://llm.mlc.ai/#ios - App Store link: https://apps.apple.com/us/app/mlc-chat/id6448482937 https://apps.apple.com/us/app/mlc-chat/id6448482937
The same team have an Android version too which I've not yet tried myself: https://llm.mlc.ai/#android https://llm.mlc.ai/#android
On my iPhone it works, and provides very decent performance. The only catch is that it needs pretty much the entire phone's memory to work - so if you switch out of it to another app and back again it resets the state and has to load the model from scratch.
- ComputerGuru 3y ago> so if you switch out of it to another app and back again it resets the state and has to load the model from scratch. This has been my experience anyway ever since upgrading to iOS 16 (and now 17) anyway. Everything is always paging, even on the latest hardware.
- doug_durham 3y agoThere is no paging in iOS. At least paging in the Linux sense. Your apps memory will get evicted when you switch to another app if the system needs it.
- dcdc123 3y agoIt is up to each app to store their state and resume. One of the most annoying apps for not doing this well is Google Maps. Holy crap I hate when I search something and get a route set and tab out to check something and com eback to a fresh Google Maps like 5 seconds later. They will fix it for a while then break it again several times a year. Once you start navigating it seems pretty reliable but until you hit start it is a shitshow.
- kergonath 3y agoThe Google Maps app is atrocious on iOS. So is their website.
- dcdc123 3y agoOther than not always storing state the Google Maps app works quite nicely on iOS.
- behnamoh 3y agoI just switched to Apple Maps and have been pretty happy with it. Uninstalled Google Maps. Telling me that in 2010 would make me laugh at you.
- dcdc123 3y agoApple Maps is terrible at discovery which is what I am doing with Google Maps 90% of the time.
- Huppie 3y agoI really want to like it, but it's pretty terrible at many things imho. I've tried really hard to switch to Apple Maps but even after three months it simply can't seem to remember I basically drive to 5-6 addresses on a weekly basis. It's always to the parking lot near the office on Monday and Wednesday morning, daycare to pick up the youngest on the way back, swimming pool on Saturday, etc. etc. but navigating to those addresses is always a pain (even selecting it from recent addresses in CarPlay) and it did sent me to the same street as the daycare in a completely different town several times now. While I value not giving more data to google, after three months I decided it simply wasn't worth it. I found the experience (especially the first time it sent me to a completely different town I never go to and I didn't catch it) pretty infuriating.
- VogonPoetry 3y agoI'd recommend setting up some Shortcuts with the addresses that can be triggered from Siri. For example, "Siri, we're going to the office", "Siri, let's go swimming", One nit with this is that I haven't been able to make it automatically transition into navigation.
- ghostpepper 3y agodo you happen to have a link where I can read more about this?
- ComputerGuru 3y agoYes, I’m aware. I shouldn’t have used that specific verbiage but that was what I meant. Which is worse than paging from a UX perspective since in practice no app does it perfectly (esp if there’s unsaved user input in a textbox).
- mi_lk 3y agoFYI that app is 3.3 GB on iOS, not your typical app size
- simonw 3y agoI'd love to know why it's that big. I'm pretty sure it doesn't include the actual LLM model files - the first time you use a model you have to click to download it, and those are likely even larger than 3.3GB.
- bm-rf 3y agoIt comes with Mistral 7B, Llama 7B and RedPajama are available for download.
- oidar 3y agoI've been playing with it on the recent iPhone Pro, and it is pretty fast. The mistral 7B is very wordy though and is prone to making up stuff. It's fun to talk to, but it's not very useful currently. Based on this demonstration of a local LLM, with optimizations, it definitely the way of the future on apple devices at least.
- keep_reading 3y agoApple has released a paper about running LLMs efficiently with low memory https://arxiv.org/pdf/2312.11514.pdf https://arxiv.org/pdf/2312.11514.pdf
- thefourthchime 3y agoMistral is amazing for its size.