4 ms·
Which LLM can run on apple's neural cores / GPU cores? I can only run on plain ol' CPU cores (llama), and it runs fine on my Ryzen CPU for less than half the pr
by beiller 3y ago
Which LLM can run on apple's neural cores / GPU cores? I can only run on plain ol' CPU cores (llama), and it runs fine on my Ryzen CPU for less than half the price of that system. That being said I'm switching from Ubuntu to Arch cause I'm sick of all my packages being way out of date!
- ingenieroariel 3y agotinygrad https://github.com/geohot/tinygrad/tree/master/accel/ane https://github.com/geohot/tinygrad/tree/master/accel/ane But I have not tested it on Linux since Asahi has not yet added support. Same machine but OSX, llama.cpp runs at 18ms per token (7B) and 200ms per token (65B) on CPU using float16.