4 ms·
Ah yes. At first, I thought that since it is all one-way forward-only communication, latency would only affect the time to the first token. But I guess the fin
by mg 2y ago
Ah yes. At first, I thought that since it is all one-way forward-only communication, latency would only affect the time to the first token.
But I guess the final output needs to be sent back to the first node before it can continue. So if there are 50 nodes with a latency of 40ms each, each token would take 2s to process.
- alexandercheema 2y agoYeah, unfortunately the autoregressive nature of these models slows it down significantly with added device<->device latency. However, you can still max out on throughput with pipeline parallelism, where you overlap execution. See: https://pytorch.org/docs/stable/pipeline.html https://pytorch.org/docs/stable/pipeline.html