I think the GPU performance for inference is probably limited currently by immaturity of PyTorch MPS (Metal) backend before I found the repo above I had a naiv

I think the GPU performance for inference is probably limited currently by immaturity of PyTorch MPS (Metal) backend before I found the repo above I had a naive attempt to get llama running with mps and it didn't "just work" - bunch of ops not supported etc