What would really be helpful here is adding information about how much inference you can do on a given GPU. Since transformers & attention scale quadratically

What would really be helpful here is adding information about how much inference you can do on a given GPU. Since transformers & attention scale quadratically this is quite important. Knowing that I can load X model on a given GPU is good, knowing that I can expect to be able to perform Y inference at N tok/sec on that GPU with however much concurrence.