3 ms·

Ehh, I’m an ML scientist with a PhD in category theory and I really don’t see it going anywhere. The comparison to geometric deep learning is especially misplac

Ehh, I’m an ML scientist with a PhD in category theory and I really don’t see it going anywhere. The comparison to geometric deep learning is especially misplaced, because I think you’ll find that Bronstein’s school sets things us as a group action, etc, but then does a lot of really hard math to actually tease out properties of how information flows through a gnn. Here they just do the Applied Category Thing and say they’ve drawn a picture so QED.

bubblyworld 2y ago
Would you be willing to elaborate a bit on that? So my understanding here is that they are using monad algebras to model the kinds of constraints you might care about (symmetry invariance/translation invariance/etc), and then by instantiating these generic monads over the category of vector spaces and fiddling through the diagrams, you recover the usual constraints on the weights in your neural net. I don't think it's supposed to work around the fact that you have to do a lot of hard math in that step, but it gives you a blueprint for doing it that generalises symmetry constraints. So you could come up with some interesting idea for a NN layer/block and use this schematic to direct your derivation of the corresponding constraints on your neural net, I guess? So it seems valuable to me in that respect, especially if they can achieve what they want (logical inference-rule-invariant NN blocks, PL semantics invariant stuff, etc). Anyway, I'm interested in your perspective/objections! (if it's technical that's fine too, I have a lot of maths background)
- catgary 2y ago
  Oh, I mean, the most obvious problem with that idea is there’s no way to ensure that whenever you update your weights that they will still satisfy those constraints. This idea has been well studied in mathematical physics, going back to Poincaré, where you work with Lie groups and Lie group actions on your action space. The reason this works, however, is you get a Lie algebra/Lie algebra action that more-or-less behaves like the tangent bundle so the same basic theory around optimization works. The main problem is they’re generalizing in the wrong direction. Everything still works when you move to Lie groupoids/Lie algebroids. You still get something like a tangent bundle, so ideas like gradient descent or Euler-Lagrange equations still make sense. Thats not the case with a generic monad - in fact the authors don’t seem to acknowledge the fact that there is some work to do regarding compatibility between the monad and derivative to ensure that gradient-based optimization will still make sense. So, basically, anyone who is familiar with the basics of optimization on manifolds or Lie groups will immediately recognize this approach as hopelessly naive. All they’ve _really_ managed to do is draw some diagrams and say “wouldn’t it be cool if these things were preserved by gradient descent.”
  - bubblyworld 2y ago
    Ah, I see, so they've basically found a nice way to express the easy stuff (finding the constraints) but the devil's in the optimisation, of course. Thanks for taking the time to reply. Coincidentally, I'm doing a project with Arnold's book on CM at the moment, so that all makes perfect sense to me.
    - catgary 2y ago
      Right, basically the generalization goes: Lie groups -> Lie group actions -> Lie groupoids. This is not a new observation (in fact Arnold’s fluid mechanics can be rephrased using lie groupoids https://tspace.library.utoronto.ca/bitstream/1807/91859/1/Fusca_Daniel_201811_PhD_thesis.pdf https://tspace.library.utoronto.ca/bitstream/1807/91859/1/Fu..., and big names like Alan Weinstein have worked in that area). I don’t think the authors actually understand that story, so they very naively went groups -> group actions -> monads. If it went that way someone in mathematical physics or optimization would have stumbled onto other concrete examples. But they haven’t, because it doesn’t.

bubblyworld 2y ago

Would you be willing to elaborate a bit on that? So my understanding here is that they are using monad algebras to model the kinds of constraints you might care about (symmetry invariance/translation invariance/etc), and then by instantiating these generic monads over the category of vector spaces and fiddling through the diagrams, you recover the usual constraints on the weights in your neural net. I don't think it's supposed to work around the fact that you have to do a lot of hard math in that step, but it gives you a blueprint for doing it that generalises symmetry constraints. So you could come up with some interesting idea for a NN layer/block and use this schematic to direct your derivation of the corresponding constraints on your neural net, I guess? So it seems valuable to me in that respect, especially if they can achieve what they want (logical inference-rule-invariant NN blocks, PL semantics invariant stuff, etc). Anyway, I'm interested in your perspective/objections! (if it's technical that's fine too, I have a lot of maths background)

catgary 2y ago
Oh, I mean, the most obvious problem with that idea is there’s no way to ensure that whenever you update your weights that they will still satisfy those constraints. This idea has been well studied in mathematical physics, going back to Poincaré, where you work with Lie groups and Lie group actions on your action space. The reason this works, however, is you get a Lie algebra/Lie algebra action that more-or-less behaves like the tangent bundle so the same basic theory around optimization works. The main problem is they’re generalizing in the wrong direction. Everything still works when you move to Lie groupoids/Lie algebroids. You still get something like a tangent bundle, so ideas like gradient descent or Euler-Lagrange equations still make sense. Thats not the case with a generic monad - in fact the authors don’t seem to acknowledge the fact that there is some work to do regarding compatibility between the monad and derivative to ensure that gradient-based optimization will still make sense. So, basically, anyone who is familiar with the basics of optimization on manifolds or Lie groups will immediately recognize this approach as hopelessly naive. All they’ve _really_ managed to do is draw some diagrams and say “wouldn’t it be cool if these things were preserved by gradient descent.”
- bubblyworld 2y ago
  Ah, I see, so they've basically found a nice way to express the easy stuff (finding the constraints) but the devil's in the optimisation, of course. Thanks for taking the time to reply. Coincidentally, I'm doing a project with Arnold's book on CM at the moment, so that all makes perfect sense to me.
  - catgary 2y ago
    Right, basically the generalization goes: Lie groups -> Lie group actions -> Lie groupoids. This is not a new observation (in fact Arnold’s fluid mechanics can be rephrased using lie groupoids https://tspace.library.utoronto.ca/bitstream/1807/91859/1/Fusca_Daniel_201811_PhD_thesis.pdf https://tspace.library.utoronto.ca/bitstream/1807/91859/1/Fu..., and big names like Alan Weinstein have worked in that area). I don’t think the authors actually understand that story, so they very naively went groups -> group actions -> monads. If it went that way someone in mathematical physics or optimization would have stumbled onto other concrete examples. But they haven’t, because it doesn’t.