3 ms·
Transformer learning explained: Coinductive guide to inductive transformer heads
- adamnemecek 4y agoI'm the author. I have recently made the realization that Hopf algebras are a surprisingly good abstraction for machine learning and this paper is an invitation for others to look into them as they can provide a unified foundation for convnets, transformers and diffusion models. Hopf algebras are a tensorial bialgebra, meaning they are both a tensor and a cotensor at once. What's the co- prefix? You can think of it as "a thing with recurrence relationships". This matters since recurrence relationships can be thought of as a generalization of autodiff. The two ML building blocks, tensors and autodiff, are then unified under one umbrella. Looking at transformers through the lens of Hopf algebra allowed me to identify their learning mechanism. Namely, the residual stream corresponds to what my paper calls unit impulse path and the attention corresponds to the convolution path of a Hopf algebra. The transformer learns by enforcing an invariant called Hopf coherence between the two paths. Hopf convolution is a generalization of the standard convolution and transformer attention emerges as a part of this convolution. Transformer models can then be understood as linear time-invariant systems. Furthermore, I believe that Hopf coherence provides a better way of doing backprop, one that occurs within the single layers as opposed to across the whole graph, but more research is needed. This approach also opens the door for verified machine learning, which I will discuss in future work. I'm currently in the process of implementing a simple machine learning framework based on Hopf algebra but it might take some time.