4 ms·
As an AI layman, I am for some reason excited by the low-bit learning models. Apple, now leaving behind the albatros that was the Apple car, could be the one t
by JKCalhoun 3y ago
As an AI layman, I am for some reason excited by the low-bit learning models.
Apple, now leaving behind the albatros that was the Apple car, could be the one to roll their own silicon and put an LLM in our back pockets.
Goodbye Siri, hello Seriously.
- talldayo 3y agoOr we could end up getting another Google integration, with iOS being too rigid and locked-down to feasibly support third-party private AI alternatives. Fate has a cruel sense of humor, sometimes.
- odyssey7 3y agoAre you referring to the Swift version of TensorFlow? https://www.infoworld.com/article/3608151/swift-for-tensorflow-project-shuts-down.html https://www.infoworld.com/article/3608151/swift-for-tensorfl...
- Me1000 3y agoI believe they're referring to these rumors: https://www.macrumors.com/2024/03/21/apple-iphone-ai-talks-google-openai-ongoing/ https://www.macrumors.com/2024/03/21/apple-iphone-ai-talks-g...
- JKCalhoun 3y agoCould be a way for Apple to buy time as well until they do have something competitive.
- deleted 3y ago[deleted]
- lawlessone 3y ago> could be the one to roll their own silicon and put an LLM in our back pockets. Wouldn't they have to do a lot of catching up with everyone else already working on it?
- declaredapple 3y agoThey've been designing their own chips a while now, including with an NPU. Also because of their unified memory design, they actually have insane bandwidth which is incredibly useful for LLMs. IMO they may have a head-start in that respect for on-device inference of large models (e.g. 1B+ params).
- talldayo 3y agoI don't think people are running 1B+ models on the Neural Engine these days. The high-performance models I've seen all rely on Metal Performance Shaders, which scales with how powerful your GPU is. It's not terribly slow on iPhone, but I think some people get the wrong idea and correlate an ambient processor like the Neural Engine with LLMs. The bigger bottleneck seems like memory, to me. iPhones have traditionally skimped on RAM moreso than even cheap and midrange Android counterparts. I can imagine running an LLM in the background on my S10 - it's a bit harder to envision iOS swapping everything smoothly on a similarly-aged iPhone.
- JKCalhoun 3y agoSure, but we're discussing 1.8-bit models that, again I'm a layman, I assume are over an order of magnitude smaller in their memory overhead.
- quentinp 3y agoActually they plan to put an LLM in your back pocket using flash memory, not silicon: https://arxiv.org/abs/2312.11514 https://arxiv.org/abs/2312.11514
- declaredapple 3y agoThe flash doesn't do the computations though, that's just a method of getting it to the processor
- sroussey 3y agoIt would be better to have eeprom or some such directly attached as memory. No loading.