I'm kinda stunned at that speed. I wonder if quantized models would affect that, i.e. GGUFs via llama.cpp. My M2 MBP would beat that, but, M2s aren't that great

I'm kinda stunned at that speed. I wonder if quantized models would affect that, i.e. GGUFs via llama.cpp. My M2 MBP would beat that, but, M2s aren't that great, like, my 3090s faster.