Yeah my explanation was definitely a lossy summary. You can do similar things with GPT, but BERT is bidirectional, so for a given token it can take into account

Yeah my explanation was definitely a lossy summary. You can do similar things with GPT, but BERT is bidirectional, so for a given token it can take into account both tokens before and after it. GPT would only take into account tokens before it. Looking both ways can be helpful. Another comment in this thread explains the same (maybe clearer).