13B runs fine on my M1 Air with 16GB Ram and 8C8G. It's not even stressing out the memory and the swap as much as Stable Diffusion. I'm getting like 350-450ms

13B runs fine on my M1 Air with 16GB Ram and 8C8G. It's not even stressing out the memory and the swap as much as Stable Diffusion. I'm getting like 350-450ms per token and it feels as fast as ChatGPT on a busy day. This obviously isn't using the Neural Engine. With Apple's Stable Diffusion implementation, when the neural engine is used I can see how my CPU and GPU stays mostly idle and the temp on the Neural Engine cores is rising but it is rising significantly less than when run on the GPU or the CPU. I wonder of it's not possible to have this run on the Neural Engine? Given that it's mostly idle, running this locally will only impact the RAM use and on a machine with a large RAM it might feel like doesn't have a performance hit and run continuously for various task.