3 ms·
The paper 'Hopfield networks are all you need' talks about why the softmax in the normal attention formulation is important for recall, and I'm always surprised
by anon291 3y ago
The paper 'Hopfield networks are all you need' talks about why the softmax in the normal attention formulation is important for recall, and I'm always surprised its ideas haven't penetrated further in the community. Basically, viewing attention as a Hopfield network, there's a theoretical maximum number of patterns that, for linear functions, is actually very low, but for the exponential functions, you get very high information density and recall.
- dilawar 3y agoTrue. And it's scaling down properties are much better than any other network I've played with (not an expert). I could run MNIST benchmark on a ESP32 board. I also liked the Convex Concave trick in the paper. The guarantee that at every step you are closer to the minima is very nice.