3 ms·
It probably helps that Apple Silicon has dedicated die space to the Neural Engine - essentially a TPU. No good for training, great for inference.
by philjohn 2y ago
It probably helps that Apple Silicon has dedicated die space to the Neural Engine - essentially a TPU. No good for training, great for inference.
- davedx 2y agoI’ve been reading up on this recently but devs say ANE is kinda a pain in the ass to leverage; most OSS is using gpu instead
- anentropic 2y agothese most likely aren't using the Neural Engine the ANE seemed to be optimised for small vision models like you might run on an iPhone a couple of years ago these will be running on the GPU
- smpanaro 2y agoI bet these can all run on ANE. I’ve run gpt2-xl 1.5B on ANE [1] and WhisperKit [2] also runs larger models on it. The smaller ones (1.1B and below) will be usably fast and with quantization I suspect the 3B one will be as well. GPU will still be faster but power for speed is the trade-off currently. [1] 7 tokens/sec https://x.com/flat/status/1719696073751400637 https://x.com/flat/status/1719696073751400637 [2] https://www.takeargmax.com/blog/whisperkit https://www.takeargmax.com/blog/whisperkit
- anentropic 2y agoindeed, but probably not as written currently? i.e they would need converting with e.g. your work in more-ane-transformers