It preserves context by taking using the whole context as input. On the section "one difference from BERT" here it is described a bit more how this works, inclu

It preserves context by taking using the whole context as input. On the section "one difference from BERT" here it is described a bit more how this works, including some nice animations. http://jalammar.github.io/illustrated-gpt2/ http://jalammar.github.io/illustrated-gpt2/ It uses both its own output from previous steps and the users prompt(s) as input for each token(word) that it predicts.