Tokens in GPT are basically short term memory, just like people have short term memory. You can't reference a million tokens back, it's been compressed into you

Tokens in GPT are basically short term memory, just like people have short term memory. You can't reference a million tokens back, it's been compressed into your long term memory. For GPT the long term memory is the trained network itself, of which it contains vastly more knowledge than any of us alone. I will say that writing perfectly working application from a prompt requires a plan, even if the output is the next word, there is a higher level plan you're not seeing in the output. This isnt sophisticated markov chains, it's understanding within the network that generates output.