3 ms·
Why would it make latency go up? It's incredibly parallelizable just copy the weights on more machines.
by naillo 4y ago
Why would it make latency go up? It's incredibly parallelizable just copy the weights on more machines.
- eachro 4y agoThe additional cpu to gpu to cpu round trip time I'm guessing?
- hooloovoo_zoo 4y agoAh the ol’ have a baby in a month by impregnating 9 women strategy.
- charcircuit 4y ago>It's incredibly parallelizable No, it isn't. It generates 1 token at a time in a loop until the response is finished. It's a highly serialized task. Parallelization increases the throughput of how many queries can be processed simultaneously, but you wouldn't be able to speed up a single query.