3 ms·
these most likely aren't using the Neural Engine the ANE seemed to be optimised for small vision models like you might run on an iPhone a couple of years ago
by anentropic 2y ago
these most likely aren't using the Neural Engine
the ANE seemed to be optimised for small vision models like you might run on an iPhone a couple of years ago
these will be running on the GPU
- smpanaro 2y agoI bet these can all run on ANE. I’ve run gpt2-xl 1.5B on ANE [1] and WhisperKit [2] also runs larger models on it. The smaller ones (1.1B and below) will be usably fast and with quantization I suspect the 3B one will be as well. GPU will still be faster but power for speed is the trade-off currently. [1] 7 tokens/sec https://x.com/flat/status/1719696073751400637 https://x.com/flat/status/1719696073751400637 [2] https://www.takeargmax.com/blog/whisperkit https://www.takeargmax.com/blog/whisperkit
- anentropic 2y agoindeed, but probably not as written currently? i.e they would need converting with e.g. your work in more-ane-transformers