3 ms·

Be sure to read the thread, in particular: https://twitter.com/goodside/status/1557926101615366144?s=21&t=6tyUdwtpHbH6TjRHYXgsIA https://twitter.com/goodside/st

Be sure to read the thread, in particular: https://twitter.com/goodside/status/1557926101615366144?s=21&t=6tyUdwtpHbH6TjRHYXgsIA https://twitter.com/goodside/status/1557926101615366144?s=21... > A caveat to all of these: I use GPT-3 a lot, so I know the “golden path” of tasks it can do reliably. Had I asked it to write a sentence backwards or sum a list of numbers, it would fail every time. These are all softball questions in isolation. I haven’t shown that GPT-3 can handle all coherent directions of this length, or even most directions that an untrained person would think to create. It’s just a demo that, if GPT-3 happens to be capable of your tasks separately, length per se is not a major issue.

sigmoid10 4y ago
>length per se is not a major issue That's kind of the whole deal of the attention mechanism in transformers and also partially why they replaced RNNs. You don't throw away any part of the original input as you construct your output. The downside is that unlike for a RNN, the total sequence length is fixed at training time and complexity grows with the square of it. But apart from computational cost, sequence length is not really an issue anymore for these models.