3 ms·

Can I run this offline on my iPhone? That would be like having basic internet search regardless of reception. Could come in handy when camping

by reidjs 3y ago

3Sophons 3y ago
You can run it on a variety of Linux, Mac and Windows based devices, including the Raspberry Pi and most laptops / servers you might have. But you still need a few GBs of memory in order to fit the model itself.
SparkyMcUnicorn 3y ago
I got this project[0] running on a Pixel. Looks like it works on some iPhones/iPads as well. [0] https://github.com/mlc-ai/mlc-llm https://github.com/mlc-ai/mlc-llm
- simonw 3y ago
  Yeah I've been using their iPhone app for a while - it works great, though it does make the phone run pretty hot while it's outputting tokens! https://llm.mlc.ai/#ios https://llm.mlc.ai/#ios
woadwarrior01 3y ago
I have a successful-ish commercial iOS app[0] for that. I'd originally built it using ggml, and then subsequently ported it to be based on mlc-llm when I found it. [0]: https://apps.apple.com/us/app/private-llm/id6448106860 https://apps.apple.com/us/app/private-llm/id6448106860
- JKCalhoun 3y ago
  Says MacOS 13 when I followed your link. Too bad I'm still on MacOS 12. (Is there a reason to require MacOS 13?)
  - woadwarrior01 3y ago
    No specific reason, but SwiftUI improved tremendously between macOS 12 and 13, and I use a couple of the newer SwiftUI features. Also, if I could go back, I’d rather not support Intel Macs. I’d built the original version of the app on an Intel Mac 6 months ago, but the performance difference between Intel Macs and Apple Silicon Macs for LLM inference with Metal is night and day. Apple won’t let me drop support for Intel Macs now, so I’ll begrudgingly support it.
    - JKCalhoun 3y ago
      Too bad about SwiftUI (not being as good on 12), but that's fair.
throwaway154 3y ago
You'd probably be better off downloading an edition of wikipedia for that purpose. Entropy, and stuff.

Can I run this offline on my iPhone? That would be like having basic internet search regardless of reception. Could come in handy when camping

by reidjs 3y ago

Can I run this offline on my iPhone? That would be like having basic internet search regardless of reception. Could come in handy when camping

3Sophons 3y ago

You can run it on a variety of Linux, Mac and Windows based devices, including the Raspberry Pi and most laptops / servers you might have. But you still need a few GBs of memory in order to fit the model itself.

SparkyMcUnicorn 3y ago

I got this project[0] running on a Pixel. Looks like it works on some iPhones/iPads as well. [0] https://github.com/mlc-ai/mlc-llm https://github.com/mlc-ai/mlc-llm

simonw 3y ago
Yeah I've been using their iPhone app for a while - it works great, though it does make the phone run pretty hot while it's outputting tokens! https://llm.mlc.ai/#ios https://llm.mlc.ai/#ios

woadwarrior01 3y ago

I have a successful-ish commercial iOS app[0] for that. I'd originally built it using ggml, and then subsequently ported it to be based on mlc-llm when I found it. [0]: https://apps.apple.com/us/app/private-llm/id6448106860 https://apps.apple.com/us/app/private-llm/id6448106860

JKCalhoun 3y ago
Says MacOS 13 when I followed your link. Too bad I'm still on MacOS 12. (Is there a reason to require MacOS 13?)
- woadwarrior01 3y ago
  No specific reason, but SwiftUI improved tremendously between macOS 12 and 13, and I use a couple of the newer SwiftUI features. Also, if I could go back, I’d rather not support Intel Macs. I’d built the original version of the app on an Intel Mac 6 months ago, but the performance difference between Intel Macs and Apple Silicon Macs for LLM inference with Metal is night and day. Apple won’t let me drop support for Intel Macs now, so I’ll begrudgingly support it.
  - JKCalhoun 3y ago
    Too bad about SwiftUI (not being as good on 12), but that's fair.

throwaway154 3y ago

You'd probably be better off downloading an edition of wikipedia for that purpose. Entropy, and stuff.