4 ms·

> Generative video models like OpenAI's Sora clearly have a world model as they are able to simulate gravity, collisions between objects, and other concepts nec

> Generative video models like OpenAI's Sora clearly have a world model as they are able to simulate gravity, collisions between objects, and other concepts necessary to render a coherent scene. I won't expand on the rest, but this is simply nonsensical. The fact that Sora generates output that matches its training data doesn't show that it has a concept of gravity, collision between object, or anything else. It has a "world model" the same way a photocopier has a "document model".

svara 2y ago
My suspicion is that you're leaving some important parts in your logic unstated. Such as belief in a magical property within humans of "understanding", which you don't define. The ability of video models to generate novel video consistent with physical reality shows that they have extracted important invariants - physical law - out of the data. It's probably better not to muddle the discussion with ill defined terms such as "intelligence" or "understanding". I have my own beef with the AGI is nigh crowd, but this criticism amounts to word play.
- phatfish 2y ago
  It feels like if these image and video generation models were really resolving some fundamental laws from the training data they should at least be able to re-create an image at a different angle.
- some1else 2y ago
  "Allegory of the cave" comes to mind, when trying to describe the understanding that's missing from diffusion models. I think a super-model with such qualifications would require a number of ControlNets in a non-visual domains to be able to encode understanding of the underlying physics. Diffusion models can render permutations of whatever they've seen fairly well without that, though.
  - svara 2y ago
    I'm very familiar with the allegory of the cave, but I'm not sure I understand where you're going with the analogy here. Are you saying that it is not possible to learn about dynamics in a higher dimensional space from a lower dimensional projection? This is clearly not true in general. E.g., video models learn that even though they're only ever seeing and outputting 2d data, objects have different sides in a fashio that is consistent with our 3d reality. The distinctions you (and others in this thread) are making is purely one of degree - how much generalization has been achieved, and how well - versus one of category.

svara 2y ago

My suspicion is that you're leaving some important parts in your logic unstated. Such as belief in a magical property within humans of "understanding", which you don't define. The ability of video models to generate novel video consistent with physical reality shows that they have extracted important invariants - physical law - out of the data. It's probably better not to muddle the discussion with ill defined terms such as "intelligence" or "understanding". I have my own beef with the AGI is nigh crowd, but this criticism amounts to word play.

phatfish 2y ago
It feels like if these image and video generation models were really resolving some fundamental laws from the training data they should at least be able to re-create an image at a different angle.
some1else 2y ago
"Allegory of the cave" comes to mind, when trying to describe the understanding that's missing from diffusion models. I think a super-model with such qualifications would require a number of ControlNets in a non-visual domains to be able to encode understanding of the underlying physics. Diffusion models can render permutations of whatever they've seen fairly well without that, though.
- svara 2y ago
  I'm very familiar with the allegory of the cave, but I'm not sure I understand where you're going with the analogy here. Are you saying that it is not possible to learn about dynamics in a higher dimensional space from a lower dimensional projection? This is clearly not true in general. E.g., video models learn that even though they're only ever seeing and outputting 2d data, objects have different sides in a fashio that is consistent with our 3d reality. The distinctions you (and others in this thread) are making is purely one of degree - how much generalization has been achieved, and how well - versus one of category.