I'm still trying to make my way through the preprint :-) EDIT: According to https://ml-jku.github.io/hopfield-layers/#update https://ml-jku.github.io/hopfield-

I'm still trying to make my way through the preprint :-) EDIT: According to https://ml-jku.github.io/hopfield-layers/#update https://ml-jku.github.io/hopfield-layers/#update , attention is the update rule for an (iterative) "dense associate memory," even though in practice it seems that one update works really, really well for attention if you train it with SGD.