GPT is an autoregressive model, this means that it's a recurring prediction model that works one "word"[*] at a time until it predicts that the text should end,

GPT is an autoregressive model, this means that it's a recurring prediction model that works one "word"[*] at a time until it predicts that the text should end, feeding it's own guesses as input for the next word. It's basically one of those markov chain bots, except with a very advanced statistical model behind it. [*] Technically it's not words, but tokens. GPT tokenizes text to better compress the amount of data being fed, it's basically a big vocabulary list that compresses text into a list of ints, for example the string "hello world" would be converted to the list [31373, 995]. For this case it's an int per word, but less common words will not be compressed this well, with the worst case scenario being a token per letter. I should also note that while the model's forward pass only works with one token at a time, the text generation is more advanced than that, there's multiple methods like beam search and top k-sampling, each with their own settings and tunables, but the gist of it is that during generation it'll try multiple combinations of token sequences and check which one is the most likely. The limitation is memory, transformer networks are notoriously memory hungry, and IIRC GPT-3 grows quadratically with the number of tokens given, usually the limit is around 2048 tokens, or roughly 1000-2000 words.