5 ms·

https://openai.com/sora?video=big-sur https://openai.com/sora?video=big-sur In this video, there's extremely consistent geometry as the camera moves, but the t

by Imnimo 3y ago

https://openai.com/sora?video=big-sur https://openai.com/sora?video=big-sur In this video, there's extremely consistent geometry as the camera moves, but the texture of the trees/shrubs on the top of the cliff on the left seems to remain very flat, reminiscent of low-poly geometry in games. I wonder if this is an artifact of the way videos are generated. Is the model separating scene geometry from camera? Maybe some sort of video-NeRF or Gaussian Splatting under the hood?

jquery 3y ago
It looks perfect to me. That's exactly how the area looks in person.
nuz 3y ago
I saw similar artifacts in dalle-1 a lot (as if the image was pasted onto geometry). Definitely wouldn't surprise me if they use synthetic rasterized data to in the training, which could totally create artifacts like this.
ethbr1 3y ago
Curious about what current SotA is on physics-infusing generation. Anyone have paper links? OpenAi has a few details: >> The current model has weaknesses. It may struggle with accurately simulating the physics of a complex scene, and may not understand specific instances of cause and effect. For example, a person might take a bite out of a cookie, but afterward, the cookie may not have a bite mark. >> Similar to GPT models, Sora uses a transformer architecture, unlocking superior scaling performance. >> We represent videos and images as collections of smaller units of data called patches, each of which is akin to a token in GPT. By unifying how we represent data, we can train diffusion transformers on a wider range of visual data than was possible before, spanning different durations, resolutions and aspect ratios. >> Sora builds on past research in DALL·E and GPT models. It uses the recaptioning technique from DALL·E 3, which involves generating highly descriptive captions for the visual training data. As a result, the model is able to follow the user’s text instructions in the generated video more faithfully. The implied facts that it understands physics of simple scenes and any instances of cause and effect are impressive! Although I assume that's been SotA-possible for awhile, and I just hadn't heard?
- msoad 3y ago
  On the announcement page, it specifically says Sora does not understand physics
deleted 3y ago
[deleted]
thomastjeffery 3y ago
The model is essentially doing nothing but dreaming. I suspect that anything that looks like familiar 3D-rendering limitations is probably a result of the training dataset simply containing a lot of actual 3D-rendered content. We can't tell a model to dream everything except extra fingers, false perspective, and 3D-rendering compromises.
- makin 3y ago
  Technically we can, that's what negative prompting[1] is about. For whatever reason, OpenAI has never exposed this capability in its image models, so it remains an open source exclusive. [1] https://stable-diffusion-art.com/how-to-use-negative-prompts/ https://stable-diffusion-art.com/how-to-use-negative-prompts...
  - thomastjeffery 3y ago
    It's more complicated than that. Negative prompts are just as limited as positive prompts.
burkaman 3y ago
Maybe it was trained on a bunch of 3d Google Earth videos.
downWidOutaFite 3y ago
Doesn't look flat to me. Edit: Here[0] I highlighted a groove in the bushes moving with perfect perspective [0] https://ibb.co/Y7WFW39 https://ibb.co/Y7WFW39
- internetter 3y ago
  Look in the top left corner, on the plane
cush 3y ago
Wow, yeah I didn't notice it at first, but looking at the rocks in the background is actually nauseating
cush 3y ago
The water is on par with Avatar. Looks perfect
spyder 3y ago
It's possible it was pre-trained on 3D renderings first, because it's easy to get almost infinite synthetic data that way, and after that they continued the training on real videos.
- iandanforth 3y ago
  In the car driving on the mountain road video you can see level-of-detail popping artifacts being reproduced, so I think that's a fair guess.
montag 3y ago
My vote is yes - some sort of intermediate representation is involved. It just seems unbelievable that it's end-to-end with 2D frames...

by Imnimo 3y ago

jquery 3y ago

It looks perfect to me. That's exactly how the area looks in person.

nuz 3y ago

I saw similar artifacts in dalle-1 a lot (as if the image was pasted onto geometry). Definitely wouldn't surprise me if they use synthetic rasterized data to in the training, which could totally create artifacts like this.

ethbr1 3y ago

Curious about what current SotA is on physics-infusing generation. Anyone have paper links? OpenAi has a few details: >> The current model has weaknesses. It may struggle with accurately simulating the physics of a complex scene, and may not understand specific instances of cause and effect. For example, a person might take a bite out of a cookie, but afterward, the cookie may not have a bite mark. >> Similar to GPT models, Sora uses a transformer architecture, unlocking superior scaling performance. >> We represent videos and images as collections of smaller units of data called patches, each of which is akin to a token in GPT. By unifying how we represent data, we can train diffusion transformers on a wider range of visual data than was possible before, spanning different durations, resolutions and aspect ratios. >> Sora builds on past research in DALL·E and GPT models. It uses the recaptioning technique from DALL·E 3, which involves generating highly descriptive captions for the visual training data. As a result, the model is able to follow the user’s text instructions in the generated video more faithfully. The implied facts that it understands physics of simple scenes and any instances of cause and effect are impressive! Although I assume that's been SotA-possible for awhile, and I just hadn't heard?

msoad 3y ago
On the announcement page, it specifically says Sora does not understand physics

deleted 3y ago

[deleted]

thomastjeffery 3y ago

The model is essentially doing nothing but dreaming. I suspect that anything that looks like familiar 3D-rendering limitations is probably a result of the training dataset simply containing a lot of actual 3D-rendered content. We can't tell a model to dream everything except extra fingers, false perspective, and 3D-rendering compromises.

makin 3y ago
Technically we can, that's what negative prompting[1] is about. For whatever reason, OpenAI has never exposed this capability in its image models, so it remains an open source exclusive. [1] https://stable-diffusion-art.com/how-to-use-negative-prompts/ https://stable-diffusion-art.com/how-to-use-negative-prompts...
- thomastjeffery 3y ago
  It's more complicated than that. Negative prompts are just as limited as positive prompts.

burkaman 3y ago

Maybe it was trained on a bunch of 3d Google Earth videos.

downWidOutaFite 3y ago

Doesn't look flat to me. Edit: Here[0] I highlighted a groove in the bushes moving with perfect perspective [0] https://ibb.co/Y7WFW39 https://ibb.co/Y7WFW39

internetter 3y ago
Look in the top left corner, on the plane

cush 3y ago

Wow, yeah I didn't notice it at first, but looking at the rocks in the background is actually nauseating

cush 3y ago

The water is on par with Avatar. Looks perfect

spyder 3y ago

It's possible it was pre-trained on 3D renderings first, because it's easy to get almost infinite synthetic data that way, and after that they continued the training on real videos.

iandanforth 3y ago
In the car driving on the mountain road video you can see level-of-detail popping artifacts being reproduced, so I think that's a fair guess.

montag 3y ago

My vote is yes - some sort of intermediate representation is involved. It just seems unbelievable that it's end-to-end with 2D frames...