4 ms·

Out of interest, how would such a system derive material properties from a still 3d scene? We know the leaves of a house plant will bow in the wind, because yo

Out of interest, how would such a system derive material properties from a still 3d scene? We know the leaves of a house plant will bow in the wind, because you have observed it in the wild, as a child several times. Even given non-still image training material, the nn would have to learn all of the physical properties of all the things from scratch, like a human. To train a physics model like this to derive the properties of material from the world is going to be bound to hilarious failures. A mountain with smoke hanging over it, is obviously a house plant.

digdugdirk 4y ago
I'd imagine you could do a frame by frame to identify points of interest and then "predict" the track they should follow into the next frame, training the model on each point of interest as you go. In this case, you wouldn't be aiming to identify the material properties specifically, but given a model that's tightly fitted enough, there should be constants that stand in for the missing material properties when it's all said and done. Just an initial thought, I'd love to hear why I'm wrong from those in the field.
- qikInNdOutReply 4y ago
  That still gives you only indicators on the physical material properties under condition x. Means, you know that with slight wind, a palm tree waves. The net still does not know, how much force it takes to break it and how the material deforms and keeps a different shape beyond certain forces. Why will a lamp post bend permanently in a hurricane, but a palmtree wont? You can derive those properties as a child, by bending a thin metal pipe and watching mums houseplant recover from a application of force. As an adult, you then project that information about thing "classifications" upon the world. Lets take another extreme, a floaties palmtree. How does on derive from the observed visual differences, that this palmtree is stiff and bouncy. Or circumstantial decorations as pysic change indicator. If there is snow on the palmtree, the tree might break in strong winds. To develop this physics reasoning is quite a larger part of childhood and i think the idea, that one can train this into a Neural Net beyond simple "all things are solid and bound to gravity" with references alone is quite a challenging endavour.
defterGoose 4y ago
I would think that what you're suggesting could be done by doing a bunch of Fourier transforms. If there is no relative motion of the camera you could assume that objects with high frequencies would be less massive than objects with low frequencies that could give you a useful bound on their mass/density. (Assuming a fairly uniform energy density in the scene) For my own curiosity, I wonder if some of the variables they're missing are based more on the context of the video, rather than the content. Like, colors, for instance. Maybe if you ran everything through in black and white you could get it closer to an integer lower bound?

digdugdirk 4y ago

I'd imagine you could do a frame by frame to identify points of interest and then "predict" the track they should follow into the next frame, training the model on each point of interest as you go. In this case, you wouldn't be aiming to identify the material properties specifically, but given a model that's tightly fitted enough, there should be constants that stand in for the missing material properties when it's all said and done. Just an initial thought, I'd love to hear why I'm wrong from those in the field.

qikInNdOutReply 4y ago
That still gives you only indicators on the physical material properties under condition x. Means, you know that with slight wind, a palm tree waves. The net still does not know, how much force it takes to break it and how the material deforms and keeps a different shape beyond certain forces. Why will a lamp post bend permanently in a hurricane, but a palmtree wont? You can derive those properties as a child, by bending a thin metal pipe and watching mums houseplant recover from a application of force. As an adult, you then project that information about thing "classifications" upon the world. Lets take another extreme, a floaties palmtree. How does on derive from the observed visual differences, that this palmtree is stiff and bouncy. Or circumstantial decorations as pysic change indicator. If there is snow on the palmtree, the tree might break in strong winds. To develop this physics reasoning is quite a larger part of childhood and i think the idea, that one can train this into a Neural Net beyond simple "all things are solid and bound to gravity" with references alone is quite a challenging endavour.

defterGoose 4y ago

I would think that what you're suggesting could be done by doing a bunch of Fourier transforms. If there is no relative motion of the camera you could assume that objects with high frequencies would be less massive than objects with low frequencies that could give you a useful bound on their mass/density. (Assuming a fairly uniform energy density in the scene) For my own curiosity, I wonder if some of the variables they're missing are based more on the context of the video, rather than the content. Like, colors, for instance. Maybe if you ran everything through in black and white you could get it closer to an integer lower bound?