Oh, I mean, the most obvious problem with that idea is there’s no way to ensure that whenever you update your weights that they will still satisfy those constra

Oh, I mean, the most obvious problem with that idea is there’s no way to ensure that whenever you update your weights that they will still satisfy those constraints. This idea has been well studied in mathematical physics, going back to Poincaré, where you work with Lie groups and Lie group actions on your action space. The reason this works, however, is you get a Lie algebra/Lie algebra action that more-or-less behaves like the tangent bundle so the same basic theory around optimization works. The main problem is they’re generalizing in the wrong direction. Everything still works when you move to Lie groupoids/Lie algebroids. You still get something like a tangent bundle, so ideas like gradient descent or Euler-Lagrange equations still make sense. Thats not the case with a generic monad - in fact the authors don’t seem to acknowledge the fact that there is some work to do regarding compatibility between the monad and derivative to ensure that gradient-based optimization will still make sense. So, basically, anyone who is familiar with the basics of optimization on manifolds or Lie groups will immediately recognize this approach as hopelessly naive. All they’ve _really_ managed to do is draw some diagrams and say “wouldn’t it be cool if these things were preserved by gradient descent.”