I'd casually like to know more about how the transformer holds onto an idea of "state" for the entities in a long text. In some loose experiments i did trying t

I'd casually like to know more about how the transformer holds onto an idea of "state" for the entities in a long text. In some loose experiments i did trying to cause confusion about the entities in the Winogrande dataset, transformer models do a great job keeping entities straight. The state representation of an entity must be more than its linguistic embedding equivalent, otherwise common English language male names like Frank and Jeff should be easier to mix up than say Frank and Sarah, or poison and knife. ....