Actually, this model uses attention layers which are kind of like a query/key=>value look-up and allow the model to merge the knowledge from neighboring time-st

Actually, this model uses attention layers which are kind of like a query/key=>value look-up and allow the model to merge the knowledge from neighboring time-steps into the current logit prediction. The result is that this model performs worse for highly repetitive words, just like humans do.