4 ms·
Apple releases eight small AI language models aimed at on-device use
- fouc 3y ago"Small Language Models" for on-device use. Neat. > The eight OpenELM models come in two flavors: four as "pretrained" (basically a raw, next-token version of the model) and four as instruction-tuned (fine-tuned for instruction following, which is more ideal for developing AI assistants and chatbots)
- ChrisArchitect 3y ago[dupe] Some more discussion: https://news.ycombinator.com/item?id=40140675 https://news.ycombinator.com/item?id=40140675
- buildbot 3y agoHuh, They used the pile - that's a pretty interesting choice for a corporate research team?
- blackeyeblitzar 3y agoThere are many variants of the pile at this point, including ones with copyrighted content removed. They are probably using one of those, like: https://huggingface.co/datasets/monology/pile-uncopyrighted https://huggingface.co/datasets/monology/pile-uncopyrighted
- buildbot 3y agoOh that’s really great to know about, thank you!
- yunohn 3y agoI don’t see any source for that from their paper. It just says a duplicate version of Pile with a citation linking to the original Pile paper itself. [15] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, An-ish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.
- GeekyBear 3y agoAs is often the case, the article gets the difference between open source and open weights wrong. Also, they don't seem to understand that Apple's variant of the MIT licence is GPL compatible. > This is Apple's variant of MIT. They've added wording around patents, which is why it gets it own shortname. Apple did not give this license a name, so we've named it "Apple MIT License". It is free and GPL compatible. https://fedoraproject.org/wiki/Licensing/Apple_MIT_License https://fedoraproject.org/wiki/Licensing/Apple_MIT_License
- gattilorenz 3y agoAlso many HN commenters don’t get the distinction between open source and open weights… not very surprising that ars technica also doesn’t
- blackeyeblitzar 3y agoI’m shocked that Ars got this wrong and is helping Apple openwash their work. But I’m seeing this same utterly basic mistake with other tech media as well, like Venture Beat: https://venturebeat.com/ai/apple-releases-openelm-small-open-source-ai-models-designed-to-run-on-device/ https://venturebeat.com/ai/apple-releases-openelm-small-open...
- Hugsun 3y agoI love this term, openwashing. It stumbles off the tongue.
- squigz 3y agoIt's invention of terms like this that make it hard for people to take some activist groups seriously...
- mensetmanusman 3y agoOpenwashthepolice
- 3y ago
- mosselman 3y agoDoes anyone know wether this will be on ollama? Is it a matter of someone caring enough to implement that or is there a technical limitation?
- ZiiS 3y agoThe Safetensors instructions https://github.com/ollama/ollama/blob/main/docs/import.md https://github.com/ollama/ollama/blob/main/docs/import.md should work.
- SushiHippie 3y agohttps://github.com/ollama/ollama/issues/3910 https://github.com/ollama/ollama/issues/3910 Currently not supported in llama.cpp, if someone has the time to check out OpenELM it probably will be implemented.
- juujian 3y agoI would be curious how much less computationally expensive these models are. Full-blown LLMs are overkill for most of the things I do with them. Does running them affect battery life of mobile devices in a major way? This could actually end up saving a ton of electricity. (Or maybe induce even more demand...)
- philjohn 3y agoIt probably helps that Apple Silicon has dedicated die space to the Neural Engine - essentially a TPU. No good for training, great for inference.
- davedx 3y agoI’ve been reading up on this recently but devs say ANE is kinda a pain in the ass to leverage; most OSS is using gpu instead
- anentropic 3y agothese most likely aren't using the Neural Engine the ANE seemed to be optimised for small vision models like you might run on an iPhone a couple of years ago these will be running on the GPU
- smpanaro 3y agoI bet these can all run on ANE. I’ve run gpt2-xl 1.5B on ANE [1] and WhisperKit [2] also runs larger models on it. The smaller ones (1.1B and below) will be usably fast and with quantization I suspect the 3B one will be as well. GPU will still be faster but power for speed is the trade-off currently. [1] 7 tokens/sec https://x.com/flat/status/1719696073751400637 https://x.com/flat/status/1719696073751400637 [2] https://www.takeargmax.com/blog/whisperkit https://www.takeargmax.com/blog/whisperkit
- anentropic 3y agoindeed, but probably not as written currently? i.e they would need converting with e.g. your work in more-ane-transformers
- winternewt 3y agoI hope somebody will take the time to incorporate a model like these into keyboard typing prediction soon.
- wildrhythms 3y agoI switched from Android to iOS and back to Android because the keyboard typing predictions on iOS were so bad in comparison. It felt like I was using a phone from 10 years ago.
- opulentegg 3y agoDid the same switch (Android -> iOS), but found SwiftKey[0] is also available on iOS. Better (or more Android:isch?) typing experience. They also solve the issue for us using multiple languages on a daily basis (first and second mother tongue + English), without having to cycle through all the iOS built-in keyboards just because you want to mix in a few English words/phrases. [0]: https://apps.apple.com/us/app/microsoft-swiftkey-ai-keyboard/id911813648 https://apps.apple.com/us/app/microsoft-swiftkey-ai-keyboard...
- deleted 3y ago[deleted]
- nunez 3y agoI thought iOS 17 was using on-device GPT-2 for keyboard predictions.
- theshrike79 3y agoApple is going exactly where I predicted they’d go. They’ve already had machine learning hardware on their phones for multiple generations, mostly “just” doing on-device image categorisation Now they’re moving into more generic LLMs, most likely supercharging a fully local Siri in either this year’s or next year’s iOS release And I'm predicting that there will be an opportunity for 3rd party developers to plug in to said LLM to provide extra data released during WWDC. So you can go "Hey Siri, what are the current news on CNN" -> CNN app provides data to Siri-language model in a standard format and it tells what's going on.
- thibauts 3y agoThe secret sauce will most likely be tight integration with Shortcuts, enabling the language interface to drive and automate existing apps.
- thethimble 3y agoI really hope that Shortcuts gets a UX overhaul. It feels so painful to write and test new shortcuts.
- wseqyrku 3y ago"play an hourly chime, oh and by the way remind me to get coffee when i'm on the way home tomorrow" no ux beats that but text/voice.
- theshrike79 3y agoShortcuts is getting to a point where I'd prefer to just write actual code instead of fighting with the stupid Scratch-style coding blocks.
- _mlbt 3y agoThat sounds a lot like AppleScript... https://en.wikipedia.org/wiki/AppleScript https://en.wikipedia.org/wiki/AppleScript I wonder how hard it would be for Apple to rebuild Shortcuts around an AppleScript backend to allow power users the ability to edit the scripts directly.
- mensetmanusman 3y agoIf they don't fix siri in the next year, I really should drop iOS. It's sad. Why cant I use my device to talk to GPT4 from the lock screen?
- littlecranky67 3y agoBecause running GPT4 for hundreds of millions iOS users is not an easy task - especially if there is no subscription model behind it.
- mensetmanusman 3y agoI subscribe to GPT-4, there is no reason I shouldn't be able to replace siri beyond Apple's fear of not making future profits.
- corv 3y agoYou can map ChatGPT to the action button and talk to it from the lock screen.
- rubymamis 3y agoDid anyone try to test any of them?
- frabcus 3y agoThis sounds like it is fully reproducible: "it also included reproducible training recipes that allow the weights (neural network files) to be replicated, which is unusual for a major tech company so far" Digging into the code, is this the best definition of the datasets used? https://github.com/apple/corenet/blob/0333b1fbb29c31809663c4e6de2654b9ff2d27de/projects/openelm/README-pretraining.md?plain=1#L58 https://github.com/apple/corenet/blob/0333b1fbb29c31809663c4... And the paper says the instruction tuning is based on UltraFeedback, this config seems to say exactly in what form: https://github.com/apple/corenet/blob/0333b1fbb29c31809663c4e6de2654b9ff2d27de/projects/openelm/instruction_tuning/openelm-instruct.yaml#L11 https://github.com/apple/corenet/blob/0333b1fbb29c31809663c4...
- resource_waste 3y agoI find it fascinating how late Apple is and how they always release some low quality products with a twist. They really give me Nintendo vibes. They can't compete on hardware, software, etc... So they put their resources into weird things that let their fanatical users say: "Apple has the best X" Since no one actually cares to have the best X, its nearly useless, but... its great for marketing and post purchase rationalization.
- Toutouxc 3y ago> They can't compete on hardware Every iPhone generation is the most powerful phone on the market.
- resource_waste 3y agoNo aux port though, cheappie phone. Also its completely unusable for my purposes due to Apple's poor security record. I have important secrets.