3 ms·
Two basic questions about the performance of the model: 1. What's the maximum context size (don't know the right term for this) that the model can consider at
by telotortium 5y ago
Two basic questions about the performance of the model:
1. What's the maximum context size (don't know the right term for this) that the model can consider at once? For example, for GPT-3 I believe the limit is around 2000 tokens (where each token corresponds to a few characters or perhaps a word in a dictionary)?
2. What is the inference latency - how long does it take to get a result from this model? Given that they don't have any demos linked to in the blog post that I can see, seems like it's long (like multiple seconds, maybe even a minute?).
- EvgeniyZh 5y agoSequence length is 2048 (page 10). Guesswork based on page 65 about compute says that single inference of 2048 tokens is about 1 petaflops, which may be not that bad if your commutinacation is fast (seconds?)
- telotortium 5y agoAccording to https://cloud.google.com/tpu https://cloud.google.com/tpu, each individual TPUv3 has 420 Teraflops, and TPUv4 is supposed to double that performance, so if that guess is correct, it should take a few seconds to do inference. Quite impressive really