4 ms·
Oh, I overlooked that! You are right. Surprising… since Apple has shown that it’s possible through CoreML (https://github.com/apple/ml-ane-transformers https://
by hannesfur 1y ago
Oh, I overlooked that! You are right. Surprising… since Apple has shown that it’s possible through CoreML (https://github.com/apple/ml-ane-transformers https://github.com/apple/ml-ane-transformers)
I would hope that the Foundation Models (https://developer.apple.com/documentation/foundationmodels https://developer.apple.com/documentation/foundationmodels) use the neural engine.
- hannesfur 1y agoEdit: Foundation Models use the Neural Engine. They are referring to a Neural Engine compatible K/V cache in this announcement: https://machinelearning.apple.com/research/introducing-apple-foundation-models https://machinelearning.apple.com/research/introducing-apple...
- fooblaster 1y agoThe neural engine not having a native programming model makes it effectively a dead end for external model development. It seems like a legacy unit that was designed for cnns with limited receptive fields, and just isn't programmable enough to be useful for the total set of models and their operators available today.
- hannesfur 1y agoThat's sadly true, over in x86 land things don't look much better in my opinion. The corresponding accelerators on modern Intel and AMD CPUs (the "Copilot PCs") are very difficult to program as well. I would love to read a blog post on someone trying though!
- fooblaster 1y agoI have a lot of the details there. Suffice to say it's a nightmare: https://www.google.com/url?sa=t&source=web&rct=j&opi=89978449&url=https://www.amd.com/content/dam/amd/en/documents/products/processors/ryzen/ai/iron-for-ryzen-ai-tutorial-micro-2024.pdf&ved=2ahUKEwiC8f-b1qaQAxWpMjQIHQ43HU8QFnoECBwQAQ&usg=AOvVaw0BYMxmtDG7G_fC1AAWyqUu https://www.google.com/url?sa=t&source=web&rct=j&opi=8997844... AMD is likely to back away from this IP relatively soon.