4 ms·
Interesting to see how bad the physics/object permanence is. I wonder if combining this with a Genie 2 type model (Google's new "world model") would be the next
by awongh 2y ago
Interesting to see how bad the physics/object permanence is. I wonder if combining this with a Genie 2 type model (Google's new "world model") would be the next step in refining it's capabilities.
- kranke155 2y agoUntil these models can figure out physics, it seems to me they will be an interesting toy
- andybak 2y agoThey can figure out a fair bit of physics. It's not a "no physics" vs "physics" thing. Rather it's a "flawed and unreliable physics" thing. It's similar to the LLM hallucination problem. LLMs produce nonsense and untruths - but they are still useful in many domains.
- Barrin92 2y agoIt's a pretty binary thing in the sense that "bad physics" pretty quickly decoheres into no physics. I saw one of these models doing a Minecraft like simulation and it looked sort of okay but then water started to end up in impossible places and once it was there it kept spreading and you ended up in some lovecraftian horror dimension. Any useful physics simluation at least needs boundary conditions to hold and these models have no boundary conditions because they have no clear categories of anything.
- kylehotchkiss 2y agoBut they don't, they just understand pixel relationships (right?)
- potatoman22 2y agoYou can model a lot of basic physics through observing 1,000,000 videos
- kranke155 2y agoNot consistently though. I think some model of understanding of physics is emergent but it doesn’t seem emergent enough. The model doesn’t understand object permanence either.
- kranke155 2y agoHere’s an idea - what if the fact that we have a body that has weight and consequence helps us understand physics? What if just visual data won’t get there because visual data lacks the sense of self? Could be interesting
- torginus 2y agoThis feels like computer graphics and the 'screen space' techniques that got introduced in the Xbox 360 generation - reflection, shadows etc. all suffered from the inability to work with off screen information and gave wildly bad answers once off screen info was required. The solution was simple - just maintain the information in world space, and sample for that. But simple does not mean cheap, and it led to a ton of redundant (as in invisible in the final image) having to be kept track of.