4 ms·
I like the Hopfield network paper. How do you think a transformer model learns? There is an overlap between Hopfield and Hopf, IIRC Ising is related to Hopf a
by adamnemecek 4y ago
I like the Hopfield network paper.
How do you think a transformer model learns?
There is an overlap between Hopfield and Hopf, IIRC Ising is related to Hopf antipode.
If you want to discuss this further, join my discord https://discord.cofunctional.ai https://discord.cofunctional.ai.
- ftxbro 4y ago> How do you think a transformer model learns? Sorry I don't know much about transformers, it's why I was reading! My small understanding is that transformers allow a kind of 'differentiable indirection' that has become practical to train now that we have these modern GPUs and automatic differentiation frameworks coupled with training algorithms.
- adamnemecek 4y agoYou are right but here's the kicker: they learn even without autodiff! So my paper tries to explain that. I argue that Hopf coherence is a better autodiff. You should peep the Pang2014 paper, I think you will understand better what's happening.
- ftxbro 4y agoIs it possible to explain what you are doing like I'm a normie who doesn't know and isn't willing to learn algebraic geometry, category theory, or control theory? In other words, explain only in terms of plain finite dimensional linear algebra like eigenvalues, left- and right-eigenvectors, row- and column-stochastic matrices, conjugate transpose, trace, matrix inverse, matrix exponential, etc.?
- adamnemecek 4y agoHopf algebras are analogous to electrical circuits with feedback and incrementally "learn" eigenvalues by processing incoming data without needing additional processing.