4 ms·

Many might miss the key paragraph at the end: "Sora serves as a foundation for models that can understand and simulate the real world, a capability we belie

Many might miss the key paragraph at the end: "Sora serves as a foundation for models that can understand and simulate the real world, a capability we believe will be an important milestone for achieving AGI." This also helps explain why the model is so good since it is trained to simulate the real world, as opposed to imitate the pixels. More importantly, its capabilities suggest AGI and general robotics could be closer than many think (even though some key weaknesses remain and further improvements are necessary before the goal is reached.) EDIT: I just saw this relevant comment by an expert at Nvidia: “If you think OpenAI Sora is a creative toy like DALLE, ... think again. Sora is a data-driven physics engine. It is a simulation of many worlds, real or fantastical. The simulator learns intricate rendering, "intuitive" physics, long-horizon reasoning, and semantic grounding, all by some denoising and gradient maths. I won't be surprised if Sora is trained on lots of synthetic data using Unreal Engine 5. It has to be! Let's breakdown the following video. Prompt: "Photorealistic closeup video of two pirate ships battling each other as they sail inside a cup of coffee." ….” https://twitter.com/DrJimFan/status/1758210245799920123 https://twitter.com/DrJimFan/status/1758210245799920123

kevmo314 3y ago
What is latent space if not a representation of the real world?
- nopinsight 3y ago
  Pretty sure many latent spaces are not trained to represent 3D motions and some detailed physics of the real world. Those in pure text LLMs, for example.
xtracto 3y ago
What I want is an AI trained to simulate the human body, allowing scientists to perform artificial human trials on all kind of medicines. Cutting trial times from years to months.
- delichon 3y ago
  Or to simulate the short or long term regret you'll feel for eating the meal in the photo.
deleted 3y ago
[deleted]
lucisferre 3y ago
> since it is trained to simulate the real world Is it though? Or is this just marketing?
- rdedev 3y ago
  If it is its not there yet. The snow in the mammoth video kind of looks like smoke, the way it rises into the air
- wilg 3y ago
  I think it's just inherent to the problem space. Obviously it understands something about the world to be able to generate convincing depictions of it.
  - lucisferre 3y ago
    It seems very dangerous to assume claims without evidence are obvious.
    - wilg 3y ago
      I didn't do that.
- deleted 3y ago
  [deleted]
- nopinsight 3y ago
  What other likely reasons might explain the leap ahead of other significant efforts? See also: https://news.ycombinator.com/item?id=39387333 https://news.ycombinator.com/item?id=39387333
  - lucisferre 3y ago
    Just having a better or bigger model? Better training data, better feedback process, etc. Seems more likely then "it can simulate reality". Also I take anecdotal reviews like that with a grain of salt. I follow numerous AI groups on Reddit and elsewhere and many users seem to have strong opinions that their tool of choice is the best. These reviews are highly biased. Not to say I'm not impressed, but it's just been released.
fasteddie31003 3y ago
Movie making is going to become fine-tuning these foundational video models. For example, if you want Brad Pitt in your movie you'll need to use his data to fine-tune his character.
mentalpiracy 3y ago
> "understand... the real world" doing a lot of heavy lifting in this statement
grbsh 3y ago
I think you are reading too far into this. The title of the technical paper is “ Video generation models as world simulators”. This is “just” a transformer that takes in a sequence of noisy image (video frame) tokens + prompt, and produces a sequence of less noisy video tokens. Repeat until noise gone. The point they’re making, which is totally valid, is that in order for such a model to produce videos with realistic physics, the underlying model is forced to learn a model of physics (a “world simulation”).
- nopinsight 3y ago
  AlphaGo and AlphaZero were able to achieve superhuman performance due to the availability of perfect simulators for the game of Go. There is no such simulator for the real world we live in. (Although pure LLMs sorta learn a rough, abstract representation of the world as perceived by humans.) Sora is an attempt to build such a simulator using deep learning. This actually affirms my comment above. “Our results suggest that scaling video generation models is a promising path towards building general purpose simulators of the physical world.” https://openai.com/research/video-generation-models-as-world-simulators https://openai.com/research/video-generation-models-as-world... What part of my argument do you disagree about?
  - lanternfish 3y ago
    `since it is trained to simulate the real world, as opposed to imitate the pixels.` It's not that its learning a model of the world instead of imitating pixels - the world model is just a necessary emergent phenomenon from the pixel imitation. It's still really impressive and very useful, but it's still 'pixel imitation'

kevmo314 3y ago

What is latent space if not a representation of the real world?

nopinsight 3y ago
Pretty sure many latent spaces are not trained to represent 3D motions and some detailed physics of the real world. Those in pure text LLMs, for example.

xtracto 3y ago

What I want is an AI trained to simulate the human body, allowing scientists to perform artificial human trials on all kind of medicines. Cutting trial times from years to months.

delichon 3y ago
Or to simulate the short or long term regret you'll feel for eating the meal in the photo.

deleted 3y ago

[deleted]

lucisferre 3y ago

> since it is trained to simulate the real world Is it though? Or is this just marketing?

rdedev 3y ago
If it is its not there yet. The snow in the mammoth video kind of looks like smoke, the way it rises into the air
wilg 3y ago
I think it's just inherent to the problem space. Obviously it understands something about the world to be able to generate convincing depictions of it.
- lucisferre 3y ago
  It seems very dangerous to assume claims without evidence are obvious.
  - wilg 3y ago
    I didn't do that.
deleted 3y ago
[deleted]
nopinsight 3y ago
What other likely reasons might explain the leap ahead of other significant efforts? See also: https://news.ycombinator.com/item?id=39387333 https://news.ycombinator.com/item?id=39387333
- lucisferre 3y ago
  Just having a better or bigger model? Better training data, better feedback process, etc. Seems more likely then "it can simulate reality". Also I take anecdotal reviews like that with a grain of salt. I follow numerous AI groups on Reddit and elsewhere and many users seem to have strong opinions that their tool of choice is the best. These reviews are highly biased. Not to say I'm not impressed, but it's just been released.

fasteddie31003 3y ago

Movie making is going to become fine-tuning these foundational video models. For example, if you want Brad Pitt in your movie you'll need to use his data to fine-tune his character.

mentalpiracy 3y ago

> "understand... the real world" doing a lot of heavy lifting in this statement

grbsh 3y ago

I think you are reading too far into this. The title of the technical paper is “ Video generation models as world simulators”. This is “just” a transformer that takes in a sequence of noisy image (video frame) tokens + prompt, and produces a sequence of less noisy video tokens. Repeat until noise gone. The point they’re making, which is totally valid, is that in order for such a model to produce videos with realistic physics, the underlying model is forced to learn a model of physics (a “world simulation”).

nopinsight 3y ago
AlphaGo and AlphaZero were able to achieve superhuman performance due to the availability of perfect simulators for the game of Go. There is no such simulator for the real world we live in. (Although pure LLMs sorta learn a rough, abstract representation of the world as perceived by humans.) Sora is an attempt to build such a simulator using deep learning. This actually affirms my comment above. “Our results suggest that scaling video generation models is a promising path towards building general purpose simulators of the physical world.” https://openai.com/research/video-generation-models-as-world-simulators https://openai.com/research/video-generation-models-as-world... What part of my argument do you disagree about?
- lanternfish 3y ago
  `since it is trained to simulate the real world, as opposed to imitate the pixels.` It's not that its learning a model of the world instead of imitating pixels - the world model is just a necessary emergent phenomenon from the pixel imitation. It's still really impressive and very useful, but it's still 'pixel imitation'