4 ms·
Hi, I work in a highly related field and I'd like to ask a few questions if you wouldn't mind. I found the work very interesting. My understanding is that essen
by jeherr 7y ago
Hi, I work in a highly related field and I'd like to ask a few questions if you wouldn't mind. I found the work very interesting. My understanding is that essentially, because of the way you have formulated the problem, you're able to add a loss to encourage the NN to learn the invarience rather than enforce it strictly. Is this correct?
I've tried implementing something somewhat related where I had a rotationally invariant learning target but was trying to use a feature vector which wasn't. I would randomly rotate my samples and add a loss function on the gradient of the rotation parameters to encourage the gradient w.r.t. rotation to be 0. Maybe in 2D this would have worked but in 3D it seemed to be too difficult for the NN to learn well enough for conservation of energy. It seems your examples use relatively simple model systems as examples. Do you have any insight into how this might work with more complex invariences?
- shoyer 7y agoNo, in this paper we enforced the Lagrangian structure is enforced as a hard constraint, via the model architecture. It's not just a loss term. Soft constraints via loss functions can also help, but in my experience they are much less effective than hard constraints. My impression is that this is pretty broadly consistent with the experience of others working in this field. For neural nets with 3D invariance, I would strongly recommend looking into the literature on "group equivariant convolutions". This has been very active area of research over the past few years, e.g., see the work of Taco Cohen: https://arxiv.org/abs/1902.04615 https://arxiv.org/abs/1902.04615
- jeherr 7y agoHmm, I must be misunderstanding the implementation somehow. Thanks for the tip. I’ll have to dig a bit deeper to understand this work here.