3 ms·

I have written a paper on this recently viewing transformers as Hopf algebras https://arxiv.org/abs/2302.01834 https://arxiv.org/abs/2302.01834. This paper is v

I have written a paper on this recently viewing transformers as Hopf algebras https://arxiv.org/abs/2302.01834 https://arxiv.org/abs/2302.01834. This paper is very relevant https://arxiv.org/abs/1206.3620 https://arxiv.org/abs/1206.3620.

Blahah 4y ago
Can you explain the implications of this for people who aren't expert in the field?
- adamnemecek 4y ago
  I argue that Hopf algebras, which are spicy tensors, provide a better autodiff and are a better model for SSMs, convnets, transformers, and diffusion models.
  - ilaksh 4y ago
    How do we use them in place of transformers? Would they allow something like the power of ChatGPT but with interpretability or modularity? Would they make it easier to create systems that behave like hybrid architectures or automatically combine different types of models (algebras?) It sounds so promising. But I have to assume that if it really is then this can be translated to code that can run on Colab or something, at least by some genius. Until it is I have to be a little skeptical and I'm not going to be able to access it anyway.
    - adamnemecek 4y ago
      Because it would show that all the models are just slightly different Hopf algebras. And yes to interpretability and modularity. It would provide guidance for building new architectures.
dontwearitout 4y ago
I've seen you pop up and recommend Hopf algebras and your paper in several deep learning threads here on HN. I am interested in your work, but find the topic very opaque and unintuitive (despite a background in physics and CS). Do you have any recommended educational resources for a gentle introduction to this?
- adamnemecek 4y ago
  Join the discord https://discord.cofunctional.ai https://discord.cofunctional.ai
  - code_biologist 4y ago
    Join the discord is a charmingly opaque answer to the question of where to find learning resources for an opaque topic. I'd guess OP was looking for something like youtube tutorials or short "programmer's guide to..." type resources. Obviously an early line of research won't have anywhere near the depth, but as an example the geometric algebra community has done a great job of this. Lots of 10 minute videos to 2 hour lectures. Steve Brunton's videos are another example "learning ML math" resource that has helped me a great deal.
    - adamnemecek 4y ago
      The thing that really boosted the GA community (I assume you mean bivector) engagement was the discord channel which was started by yours truly. You can check out the references in my paper.
      - code_biologist 4y ago
        Then good luck building community around Hopf algebras, you certainly know the roadmap better than I. The internet is an amazing place to spread word on better mathematical tools. Edit: I also just found https://bivector.net/ https://bivector.net/ Holy crap. So good.
  - Buttons840 4y ago

Blahah 4y ago

Can you explain the implications of this for people who aren't expert in the field?

adamnemecek 4y ago
I argue that Hopf algebras, which are spicy tensors, provide a better autodiff and are a better model for SSMs, convnets, transformers, and diffusion models.
- ilaksh 4y ago
  How do we use them in place of transformers? Would they allow something like the power of ChatGPT but with interpretability or modularity? Would they make it easier to create systems that behave like hybrid architectures or automatically combine different types of models (algebras?) It sounds so promising. But I have to assume that if it really is then this can be translated to code that can run on Colab or something, at least by some genius. Until it is I have to be a little skeptical and I'm not going to be able to access it anyway.
  - adamnemecek 4y ago
    Because it would show that all the models are just slightly different Hopf algebras. And yes to interpretability and modularity. It would provide guidance for building new architectures.

dontwearitout 4y ago

I've seen you pop up and recommend Hopf algebras and your paper in several deep learning threads here on HN. I am interested in your work, but find the topic very opaque and unintuitive (despite a background in physics and CS). Do you have any recommended educational resources for a gentle introduction to this?

adamnemecek 4y ago
Join the discord https://discord.cofunctional.ai https://discord.cofunctional.ai
- code_biologist 4y ago
  Join the discord is a charmingly opaque answer to the question of where to find learning resources for an opaque topic. I'd guess OP was looking for something like youtube tutorials or short "programmer's guide to..." type resources. Obviously an early line of research won't have anywhere near the depth, but as an example the geometric algebra community has done a great job of this. Lots of 10 minute videos to 2 hour lectures. Steve Brunton's videos are another example "learning ML math" resource that has helped me a great deal.
  - adamnemecek 4y ago
    The thing that really boosted the GA community (I assume you mean bivector) engagement was the discord channel which was started by yours truly. You can check out the references in my paper.
    - code_biologist 4y ago
      Then good luck building community around Hopf algebras, you certainly know the roadmap better than I. The internet is an amazing place to spread word on better mathematical tools. Edit: I also just found https://bivector.net/ https://bivector.net/ Holy crap. So good.
- Buttons840 4y ago