4 ms·

I wonder if Apple will have anything to show with on-device LLMs at WWDC in a couple weeks, or if that's a feature for another year? Definitely feels like some

I wonder if Apple will have anything to show with on-device LLMs at WWDC in a couple weeks, or if that's a feature for another year? Definitely feels like something where a lot of apps will want to take advantage of them, and there's some amount of space saving and small developer approachability from building one in as a system service, rather than everybody bringing their own.

dghlsakjg 3y ago
I have my doubts about them opening up their own LLMs. But it wouldn’t surprise me to see them talking about what their m-chips can run onboard. Although, it would be pretty cool to run Siri locally. And also if she didn’t suck
- wlesieutre 3y ago
  Some of Siri runs locally now, much better for setting timers than it used to be. But still way to much of > Hey Siri, set a timer for 5 minutes > I set your "Timer" alarm for 5:00 AM
  - tailspin2019 3y ago
    And this... > Hey Siri, set a reminder for next Tuesday at 11am; "Remember John arriving tomorrow" > Ok, I set your reminder for tomorrow for "Remember John arriving" ... > Hey Siri, change it to next Tuesday at 11am > Ok, your reminder for tomorrow is now called "Next Tuesday at 11am"
valine 3y ago
Useful LLMs are probably still a tad to large to run locally on device. Loading and unloading the model takes several seconds, so ideally the model would just sit in RAM at all times. In my experience playing with local LLaMA variants, ChatGPT level responses are really only possible with 13B parameter models and up. With 4bit quantization that's 6.5GB of RAM usage just for the model weights. You could totally run that on a phone today, but you'll be getting 2 tokens/s and using the majority of the phone's RAM. A responsive, local LLM on a phone is going to happen someday soon, but we need some serious hardware improvements first. I'd speculate you'll need at least 24GB of shared memory, and considerably more dedicated tensor multiplication cores. Alternatively, some sort of GPT architecture change that reduces the memory overhead could also drive local LLMs. Who knows, maybe Apple has a trick or two up their sleeves.

dghlsakjg 3y ago

I have my doubts about them opening up their own LLMs. But it wouldn’t surprise me to see them talking about what their m-chips can run onboard. Although, it would be pretty cool to run Siri locally. And also if she didn’t suck

wlesieutre 3y ago
Some of Siri runs locally now, much better for setting timers than it used to be. But still way to much of > Hey Siri, set a timer for 5 minutes > I set your "Timer" alarm for 5:00 AM
- tailspin2019 3y ago
  And this... > Hey Siri, set a reminder for next Tuesday at 11am; "Remember John arriving tomorrow" > Ok, I set your reminder for tomorrow for "Remember John arriving" ... > Hey Siri, change it to next Tuesday at 11am > Ok, your reminder for tomorrow is now called "Next Tuesday at 11am"

valine 3y ago

Useful LLMs are probably still a tad to large to run locally on device. Loading and unloading the model takes several seconds, so ideally the model would just sit in RAM at all times. In my experience playing with local LLaMA variants, ChatGPT level responses are really only possible with 13B parameter models and up. With 4bit quantization that's 6.5GB of RAM usage just for the model weights. You could totally run that on a phone today, but you'll be getting 2 tokens/s and using the majority of the phone's RAM. A responsive, local LLM on a phone is going to happen someday soon, but we need some serious hardware improvements first. I'd speculate you'll need at least 24GB of shared memory, and considerably more dedicated tensor multiplication cores. Alternatively, some sort of GPT architecture change that reduces the memory overhead could also drive local LLMs. Who knows, maybe Apple has a trick or two up their sleeves.