> Isn't this memorization at the end? That would be overfitting and it's a known issue you're trying to avoid during training. > How can we correct for this i

> Isn't this memorization at the end? That would be overfitting and it's a known issue you're trying to avoid during training. > How can we correct for this in the transformer architecture? I don't think the post really answers whether we need to. It may be just a case of this type of idea not being well represented in the training data, so didn't generalise during training.