4 ms·

This reminded me a great deal of a paper I read like 10 years ago, which I just looked up because I wanted to see if it was cited here, "Distilling Free-Form Na

This reminded me a great deal of a paper I read like 10 years ago, which I just looked up because I wanted to see if it was cited here, "Distilling Free-Form Natural Laws from Experimental Data" [0]. Lo and behold it is not only cited, but is written by the same authors. So the novelty here, as discussed in the paper [1], is that they are doing something similar but from video instead of from sensor streams. Which is quite interesting, as it opens up the available information to systems that are hard to 'sense' apart from pointing a camera at them, like the lava lamp example. I did see a presentation a few years ago on a similar topic by Erwin Couwans, author of Bullet physics engine, who was discussing doing neural inference of physics. He basically wanted to replace Bullet with a neural network, which I thought was kind of funny at the time, but cool if it worked. (Physical laws at this level of mechanics are mostly quite well understood, but the devil is in the details and actually solving time-discretized non-smooth dynamics, as well as performing good contact detection is less than obvious the more you get into it, not to mention friction, so I could see why a "learned" solution could be attractive, but at the time I found it funny that he was proposing to learn the physics that were already simulated by Bullet.) Looking back, it appears that those ideas culminated in a paper [2] two years ago, which I'll have to read now -- from the abstract, the differentiability of the simulation has benefits not only for learning the physics (mostly friction models it appears), but for learning controllers for plants within the simulated physics. It seems to be not cited in the linked article but maybe it is a slightly different, but related, topic. In any case, fascinating stuff. [0]: https://www.science.org/doi/10.1126/science.1165893 https://www.science.org/doi/10.1126/science.1165893 [1]: https://arxiv.org/abs/2112.10755 https://arxiv.org/abs/2112.10755 [2]: https://arxiv.org/abs/2011.04217 https://arxiv.org/abs/2011.04217

ok_dad 4y ago
That’s interesting thanks for sharing. I always was bearish on neural networks when advertised as “AI”, but now I think that perhaps when used as a tool rather than a panacea they could be very useful. Humans learn to operate heavy machinery in industrial settings through experience and use via our own neural networks, and we have incomplete understanding of physics, so why not use a neural network to learn the physically best way to drive an industrial machine and use the physics as backup for safety? Seems to work well based on all these stories so far, so maybe learning neural networks are the next differential equations for engineering.
- radarsat1 4y ago
  The trick is to remember that a neural network is a function approximator. A good deal of AI research is in the business of casting "intelligence" as a "function" so that you can pose a problem and figure out how to feed it data, ie., well-defined input and output. That's why different function approximators can be used for AI, such as decision trees, etc., not just neural networks. What to model, and how to model it, are orthogonal problems, both interesting in their own right. It just happens that NN are particularly good at handling high-dimensional inputs, which are necessary for perception tasks, as well as for handling large vocabulary language modeling, so they are doing rather well lately. On the other hand, there are lots of places where "functions" are useful, that have nothing to do with "intelligence", but where actually writing the function with full fidelity can be difficult or intractable. These are also opportunities where learnable function approximation can provide some great benefit, provided you can figure out how to pose the problem such that data is available for the learning part. A good example is in physically-based rendering, you have lots of complicated aspects of light transport for which we can model the physics quite well, but when you get into complicated reflections and scattering, etc., this can all be modeled by a complicated function called the BRDF [0]. Hand-writing a good BRDF is possible, and quite typical of high-end renderers in fact, but it's no surprise that there's been research in replacing it with a "neural BRDF" [e.g. 1]. That's just to give an example of a place where a single, very targeted and small (but complicated) part of a larger framework can stand to benefit from data-driven modeling, and a neural network can be one good way to do that. Another example is similar usage in computation fluid dynamics [2], where we can hand write a pretty good model, but to capture what is missing, it can be useful to have an approximator. The problem there is that having approximated the function well, it doesn't necessarily lead to better human understanding of the phenomenon. Discovering the true, sparse latent variables in a way that is interpretable, instead of a black box, is a useful step towards that. (Which I guess the current article is aiming at, but I haven't read it in full yet.) But sometimes all you want is results, as in the case of synthesizing a good controller. For example in CFD, if you can use the blackbox model to generate a good stabilizer for an ocean platform, you don't really care about the physics, as long as it's accurate enough to be trustworthy. So the utility of these methods, like most things, is relative to what your goals are. [0]: https://pbr-book.org/3ed-2018/Color_and_Radiometry/Surface_Reflection https://pbr-book.org/3ed-2018/Color_and_Radiometry/Surface_R... [1]: https://arxiv.org/abs/2111.03797 https://arxiv.org/abs/2111.03797 [2]: https://github.com/loliverhennigh/Computational-Fluid-Dynamics-Machine-Learning-Examples https://github.com/loliverhennigh/Computational-Fluid-Dynami...
qikInNdOutReply 4y ago
Out of interest, how would such a system derive material properties from a still 3d scene? We know the leaves of a house plant will bow in the wind, because you have observed it in the wild, as a child several times. Even given non-still image training material, the nn would have to learn all of the physical properties of all the things from scratch, like a human. To train a physics model like this to derive the properties of material from the world is going to be bound to hilarious failures. A mountain with smoke hanging over it, is obviously a house plant.
- digdugdirk 4y ago
  I'd imagine you could do a frame by frame to identify points of interest and then "predict" the track they should follow into the next frame, training the model on each point of interest as you go. In this case, you wouldn't be aiming to identify the material properties specifically, but given a model that's tightly fitted enough, there should be constants that stand in for the missing material properties when it's all said and done. Just an initial thought, I'd love to hear why I'm wrong from those in the field.
  - qikInNdOutReply 4y ago
    That still gives you only indicators on the physical material properties under condition x. Means, you know that with slight wind, a palm tree waves. The net still does not know, how much force it takes to break it and how the material deforms and keeps a different shape beyond certain forces. Why will a lamp post bend permanently in a hurricane, but a palmtree wont? You can derive those properties as a child, by bending a thin metal pipe and watching mums houseplant recover from a application of force. As an adult, you then project that information about thing "classifications" upon the world. Lets take another extreme, a floaties palmtree. How does on derive from the observed visual differences, that this palmtree is stiff and bouncy. Or circumstantial decorations as pysic change indicator. If there is snow on the palmtree, the tree might break in strong winds. To develop this physics reasoning is quite a larger part of childhood and i think the idea, that one can train this into a Neural Net beyond simple "all things are solid and bound to gravity" with references alone is quite a challenging endavour.
t_mann 4y ago
Thanks for those links! Back in school, I loved wondering about whether things like the inverse square law for gravity could be (have been) discovered from raw experimental data via ML - cool to see that it's actually been done already.
dls2016 4y ago
Piggy backing here... but this dimension-reduction or system-identification stuff always reminds me of Takens' theorem [0]. He even did some neural network stuff back in the "dark ages" of the topic [1]. [0] https://en.wikipedia.org/wiki/Takens%27s_theorem https://en.wikipedia.org/wiki/Takens%27s_theorem [1] https://clgiles.ist.psu.edu/papers/NC-2000-learning-chaos-nn.pdf https://clgiles.ist.psu.edu/papers/NC-2000-learning-chaos-nn...
- adamnemecek 4y ago
  This is not surprising considering this is just a fixed point or diagonalization. I have written up a bit on this topic https://github.com/adamnemecek/adjoint https://github.com/adamnemecek/adjoint
  - dls2016 4y ago
    Did you mean to reply to my other post? If so, you may be interested in: https://en.wikipedia.org/wiki/Pontryagin_duality https://en.wikipedia.org/wiki/Pontryagin_duality

ok_dad 4y ago

That’s interesting thanks for sharing. I always was bearish on neural networks when advertised as “AI”, but now I think that perhaps when used as a tool rather than a panacea they could be very useful. Humans learn to operate heavy machinery in industrial settings through experience and use via our own neural networks, and we have incomplete understanding of physics, so why not use a neural network to learn the physically best way to drive an industrial machine and use the physics as backup for safety? Seems to work well based on all these stories so far, so maybe learning neural networks are the next differential equations for engineering.

radarsat1 4y ago
The trick is to remember that a neural network is a function approximator. A good deal of AI research is in the business of casting "intelligence" as a "function" so that you can pose a problem and figure out how to feed it data, ie., well-defined input and output. That's why different function approximators can be used for AI, such as decision trees, etc., not just neural networks. What to model, and how to model it, are orthogonal problems, both interesting in their own right. It just happens that NN are particularly good at handling high-dimensional inputs, which are necessary for perception tasks, as well as for handling large vocabulary language modeling, so they are doing rather well lately. On the other hand, there are lots of places where "functions" are useful, that have nothing to do with "intelligence", but where actually writing the function with full fidelity can be difficult or intractable. These are also opportunities where learnable function approximation can provide some great benefit, provided you can figure out how to pose the problem such that data is available for the learning part. A good example is in physically-based rendering, you have lots of complicated aspects of light transport for which we can model the physics quite well, but when you get into complicated reflections and scattering, etc., this can all be modeled by a complicated function called the BRDF [0]. Hand-writing a good BRDF is possible, and quite typical of high-end renderers in fact, but it's no surprise that there's been research in replacing it with a "neural BRDF" [e.g. 1]. That's just to give an example of a place where a single, very targeted and small (but complicated) part of a larger framework can stand to benefit from data-driven modeling, and a neural network can be one good way to do that. Another example is similar usage in computation fluid dynamics [2], where we can hand write a pretty good model, but to capture what is missing, it can be useful to have an approximator. The problem there is that having approximated the function well, it doesn't necessarily lead to better human understanding of the phenomenon. Discovering the true, sparse latent variables in a way that is interpretable, instead of a black box, is a useful step towards that. (Which I guess the current article is aiming at, but I haven't read it in full yet.) But sometimes all you want is results, as in the case of synthesizing a good controller. For example in CFD, if you can use the blackbox model to generate a good stabilizer for an ocean platform, you don't really care about the physics, as long as it's accurate enough to be trustworthy. So the utility of these methods, like most things, is relative to what your goals are. [0]: https://pbr-book.org/3ed-2018/Color_and_Radiometry/Surface_Reflection https://pbr-book.org/3ed-2018/Color_and_Radiometry/Surface_R... [1]: https://arxiv.org/abs/2111.03797 https://arxiv.org/abs/2111.03797 [2]: https://github.com/loliverhennigh/Computational-Fluid-Dynamics-Machine-Learning-Examples https://github.com/loliverhennigh/Computational-Fluid-Dynami...

qikInNdOutReply 4y ago

Out of interest, how would such a system derive material properties from a still 3d scene? We know the leaves of a house plant will bow in the wind, because you have observed it in the wild, as a child several times. Even given non-still image training material, the nn would have to learn all of the physical properties of all the things from scratch, like a human. To train a physics model like this to derive the properties of material from the world is going to be bound to hilarious failures. A mountain with smoke hanging over it, is obviously a house plant.

digdugdirk 4y ago
I'd imagine you could do a frame by frame to identify points of interest and then "predict" the track they should follow into the next frame, training the model on each point of interest as you go. In this case, you wouldn't be aiming to identify the material properties specifically, but given a model that's tightly fitted enough, there should be constants that stand in for the missing material properties when it's all said and done. Just an initial thought, I'd love to hear why I'm wrong from those in the field.
- qikInNdOutReply 4y ago
  That still gives you only indicators on the physical material properties under condition x. Means, you know that with slight wind, a palm tree waves. The net still does not know, how much force it takes to break it and how the material deforms and keeps a different shape beyond certain forces. Why will a lamp post bend permanently in a hurricane, but a palmtree wont? You can derive those properties as a child, by bending a thin metal pipe and watching mums houseplant recover from a application of force. As an adult, you then project that information about thing "classifications" upon the world. Lets take another extreme, a floaties palmtree. How does on derive from the observed visual differences, that this palmtree is stiff and bouncy. Or circumstantial decorations as pysic change indicator. If there is snow on the palmtree, the tree might break in strong winds. To develop this physics reasoning is quite a larger part of childhood and i think the idea, that one can train this into a Neural Net beyond simple "all things are solid and bound to gravity" with references alone is quite a challenging endavour.

t_mann 4y ago

Thanks for those links! Back in school, I loved wondering about whether things like the inverse square law for gravity could be (have been) discovered from raw experimental data via ML - cool to see that it's actually been done already.

dls2016 4y ago

Piggy backing here... but this dimension-reduction or system-identification stuff always reminds me of Takens' theorem [0]. He even did some neural network stuff back in the "dark ages" of the topic [1]. [0] https://en.wikipedia.org/wiki/Takens%27s_theorem https://en.wikipedia.org/wiki/Takens%27s_theorem [1] https://clgiles.ist.psu.edu/papers/NC-2000-learning-chaos-nn.pdf https://clgiles.ist.psu.edu/papers/NC-2000-learning-chaos-nn...

adamnemecek 4y ago
This is not surprising considering this is just a fixed point or diagonalization. I have written up a bit on this topic https://github.com/adamnemecek/adjoint https://github.com/adamnemecek/adjoint
- dls2016 4y ago
  Did you mean to reply to my other post? If so, you may be interested in: https://en.wikipedia.org/wiki/Pontryagin_duality https://en.wikipedia.org/wiki/Pontryagin_duality