4 ms·
I have recently written a paper on understanding transformer learning via the lens of coinduction & Hopf algebra. https://arxiv.org/abs/2302.01834v1 https://ar
by adamnemecek 4y ago
I have recently written a paper on understanding transformer learning via the lens of coinduction & Hopf algebra.
https://arxiv.org/abs/2302.01834v1 https://arxiv.org/abs/2302.01834v1
The learning mechanism of transformer models was poorly understood however it turns out that a transformer is like a circuit with a feedback.
I argue that autodiff can be replaced with what I call in the paper Hopf coherence.
Furthermore, if we view transformers as Hopf algebras, one can bring convolutional models, diffusion models and transformers under a single umbrella.
I'm working on a next gen Hopf algebra based machine learning framework.
Join my discord if you want to discuss this further https://discord.gg/mr9TAhpyBW https://discord.gg/mr9TAhpyBW
- erichocean 4y ago> Furthermore, if we view transformers as Hopf algebras, one can bring convolutional models, diffusion models and transformers under a single umbrella. Have you written any more about this?
- adamnemecek 4y agoLook into the connection between diffusion and Hopf algebras.
- mi_ainsel 4y agoApologies if I'm missing something completely obvious but I'm having some difficulties dredging up material that connects Hopf algebras and diffusion. Are there any resources that you would recommend in addition to your paper?
- amkkma 4y agoIs there any hope of understanding this with just calc and linalg knowledge?
- adamnemecek 4y agoI think so. The main idea is the idea of Hopf coherence. The transformer/Hopf algebra update their internal state in order to enforce the Hopf coherence formula (you can find that in the paper). The idea of streams (as in infinite lists) is related to this via coalgebras.
- erichocean 4y agoHi Adam, can you update your Discord invite? It's now invalid.