53 ms·
They are pretty slow even on GPU. The problem is that it's an autoregressive model. So it needs to do a forward pass for each token.
by visarga 4y ago
They are pretty slow even on GPU. The problem is that it's an autoregressive model. So it needs to do a forward pass for each token.