3 ms·

I'd love to have my iPhone communicate with a Mac Studio at my house, for the heavy lifting. I realize this would be slower than having on-device processing, bu

I'd love to have my iPhone communicate with a Mac Studio at my house, for the heavy lifting. I realize this would be slower than having on-device processing, but it would be much better for battery life. And although I trust Apple's privacy more than Google/FB, I'd still rather keep my AI interactions separate from anyone's cloud.

jayd16 3y ago
> it would be much better for battery life. I wonder what the numbers actually are for local compute on custom hardware compared to firing up the wifi antennae to make the remote request.
- gnicholas 3y ago
  Yeah I have wondered about this. But seeing how an LLM hammers my M2 MBA CPU for many seconds per request, I’m guessing this would have a significant impact on a smartphone battery.
  - gcr 3y ago
    ANE power usage for always-on Hey Siri wake word detection is impressively low. The language models we have are orders of magnitude way too big, but they won’t be for long. I think we’ll be surprised. For comparison, a quantized mobilenetv2 takes 3.7MB disk space to solve the same image understanding task as the old VGG Caffe models which take 550MB. We’ve come a long way in six years.
- circuit10 3y ago
  Good language models are NOT easy to run, so I’d imagine local compute would take many orders of magnitude more power
smoldesu 3y ago
You might be pleased to hear that nothing really stops you from doing this today. If you ran Serge[0] on a Mac with Tailscale, you could hack together a decently-accelerated Llama chatbot. [0] https://github.com/serge-chat/serge https://github.com/serge-chat/serge
- gnicholas 3y ago
  I'm not technical enough to be able to hack this together, but I do hope that enough other people have the same itch, and are able to scratch it!
bob-09 3y ago
I’d love to see both options and a seamless transition between them. An option tuned for home use that utilizes the processing power of home devices and local networks, and an option tuned for on-the-go use utilizing the iPhone/iPad processors and mobile networks.

jayd16 3y ago

> it would be much better for battery life. I wonder what the numbers actually are for local compute on custom hardware compared to firing up the wifi antennae to make the remote request.

gnicholas 3y ago
Yeah I have wondered about this. But seeing how an LLM hammers my M2 MBA CPU for many seconds per request, I’m guessing this would have a significant impact on a smartphone battery.
- gcr 3y ago
  ANE power usage for always-on Hey Siri wake word detection is impressively low. The language models we have are orders of magnitude way too big, but they won’t be for long. I think we’ll be surprised. For comparison, a quantized mobilenetv2 takes 3.7MB disk space to solve the same image understanding task as the old VGG Caffe models which take 550MB. We’ve come a long way in six years.
circuit10 3y ago
Good language models are NOT easy to run, so I’d imagine local compute would take many orders of magnitude more power

smoldesu 3y ago

You might be pleased to hear that nothing really stops you from doing this today. If you ran Serge[0] on a Mac with Tailscale, you could hack together a decently-accelerated Llama chatbot. [0] https://github.com/serge-chat/serge https://github.com/serge-chat/serge

gnicholas 3y ago
I'm not technical enough to be able to hack this together, but I do hope that enough other people have the same itch, and are able to scratch it!

bob-09 3y ago

I’d love to see both options and a seamless transition between them. An option tuned for home use that utilizes the processing power of home devices and local networks, and an option tuned for on-the-go use utilizing the iPhone/iPad processors and mobile networks.