Matrix multiply scales polynomially so don't be thrown off by that. You'd also expect some increases in cost where things move from single GPU to multi-GPU to

Matrix multiply scales polynomially so don't be thrown off by that. You'd also expect some increases in cost where things move from single GPU to multi-GPU to networked machines. I'm still not sure if that pushes things to exponential or would just step up cost at each transition.