This blogger implies that GPT-3 could be within an order of magnitude of reaching a threshold of data efficiency. The implication is that a GPT model with >1 tr

This blogger implies that GPT-3 could be within an order of magnitude of reaching a threshold of data efficiency. The implication is that a GPT model with >1 trillion parameters would begin to see a reduction in data efficiency. I have read all the GPT related papers and I’m frankly not sure what nostalgebraist thinks this would mean, practically speaking. All complex problem domains begin to see a dropoff in data efficiency once the “easy” structure is successfully learned. The lesson here might simply be that GPT type models are close (within an order of magnitude, so “close” is subjective) to being able to learn all the “obvious” regularities in massive language datasets, leaving the increasingly subtle regularities that might require very specific, very difficult-to-reach-through-local-minima-ravines, or very “big” learned abstractions to discover. Since GPT-3 can already do some rather incredible things in the zero-shot case, to say nothing of the few-shot case, this fails to make me feel suddenly dissatisfied with the performance large transformer models.