Yeah, unfortunately the autoregressive nature of these models slows it down significantly with added device<->device latency. However, you can still max out on

Yeah, unfortunately the autoregressive nature of these models slows it down significantly with added device<->device latency. However, you can still max out on throughput with pipeline parallelism, where you overlap execution. See: https://pytorch.org/docs/stable/pipeline.html https://pytorch.org/docs/stable/pipeline.html