It makes no sense to make "x10", or "x40" speedup claims, without mentioning a specific benchmark. I'm guessing the code that scales well with 200 or even 12 G

It makes no sense to make "x10", or "x40" speedup claims, without mentioning a specific benchmark. I'm guessing the code that scales well with 200 or even 12 GPUs is fairly rare. For example, problems like ImageNet classification with convolutional NNs don't scale well beyond 2-4 GPUs. At some point, you will actually get a slowdown, because of the communication overhead.