3 ms·
tinygrad https://github.com/geohot/tinygrad/tree/master/accel/ane https://github.com/geohot/tinygrad/tree/master/accel/ane But I have not tested it on Linux s
by ingenieroariel 4y ago
tinygrad
https://github.com/geohot/tinygrad/tree/master/accel/ane https://github.com/geohot/tinygrad/tree/master/accel/ane
But I have not tested it on Linux since Asahi has not yet added support.
Same machine but OSX, llama.cpp runs at 18ms per token (7B) and 200ms per token (65B) on CPU using float16.