14 ms·

Video generation models as world simulators

361 points by linksbro 3y ago

lairv 3y ago
I find it wild that this model does not have explicit 3D prior, yet learns to generate videos with such 3D consistency, you can directly train a 3D representation (NeRF-like) from those videos: https://twitter.com/BenMildenhall/status/1758224827788468722 https://twitter.com/BenMildenhall/status/1758224827788468722
- nodja 3y ago
  This is also true for 2D diffusion models[1]. I suppose they need to understand how 3d works for stuff like lighting/shadows/object occlusion, etc. [1] https://dreamfusion3d.github.io/ https://dreamfusion3d.github.io/
- crooked-v 3y ago
  That leaves me wondering if it'd be possible to get some variant of the model to directly output 3D meshes and camera animation instead of an image.
- Nihilartikel 3y ago
  I was similarly astonished at this adaptation of stable diffusion to make HDR spherical environment maps from existing images- https://diffusionlight.github.io/ https://diffusionlight.github.io/ The crazy thing is that they do it by prompting the model to in paint a chrome sphere into the middle of the image to reflect what is behind the camera! The model can interpret the context and dream up what is plausibly in the whole environment.
  - emadm 3y ago
    Yeah we were surprised by that, video models are great 3d prices and image models are great video model priors
- TOMDM 3y ago
  I wonder how much it'd improve if trained on stereo image data.
  - QuadmasterXLII 3y ago
    Moving camera is just stereo.
- larschdk 3y ago
  You aren't looking carefully enough. I find so many inconsistencies in these examples. Perspectives that are completely wrong when the camera rotates. Windows that shift perspective, patios that are suddenly deep/shallow. Shadows that appear/disappear as the camera shifts. In other examples; paths, objects, people suddenly appearing or disappearing out of nowhere. A stone turning into a person. A horse that suddenly has a second head, then becomes a separate horse with only two legs. It is impressive at a glance, but if you pay attention, it is more like dreaming than realism (images conjured out of other images, without attention to long term temporal, spatial, and causal consistency). I'm hardly more impressed that Google's deep dream, which is 10 years old.
  - beezlebroxxxxxx 3y ago
    Yeah, it seems to have a hard time processing lens distortion in particular which gives a very weird quality. It's actually bending things, or trying to fill in the gaps, instead of distorting the image in the "correct" way.
  - lairv 3y ago
    You can literally run 3D algorithms like NeRF or COLMAP on those videos (check the tweet I sent), it's not my opinion, those videos are sufficiently 3D consistent that you can extract 3D geometry from them Surely it's not perfect, but this was not the case for previous video generation algorithms
data-ottawa 3y ago
I know the main post has been getting a lot of reaction, but this page absolutely blew me away. The results are striking. The robot examples are very underwhelming, but the people and background people are all very well done, and at a level much better than most static image diffusion models produce. Generating the same people as the interact with objects is also not something I expected a model like this to do well so soon.
colesantiago 3y ago
Damn, even minecraft videos being simulated, this is crazy to see from OpenAI. Edit, changed the links to the direct ones! https://cdn.openai.com/tmp/s/simulation_6.mp4 https://cdn.openai.com/tmp/s/simulation_6.mp4 https://cdn.openai.com/tmp/s/simulation_7.mp4 https://cdn.openai.com/tmp/s/simulation_7.mp4
- SushiHippie 3y ago
  But it starts to make sense, when you think about the fact that Minecraft is owned by Microsoft. example video links from TFA: https://cdn.openai.com/tmp/s/simulation_6.mp4 https://cdn.openai.com/tmp/s/simulation_6.mp4 https://cdn.openai.com/tmp/s/simulation_7.mp4 https://cdn.openai.com/tmp/s/simulation_7.mp4
  - empath-nirvana 3y ago
    I don't think that has anything to do with it. There's just millions of hours of minecraft video on youtube and twitch.
    - SushiHippie 3y ago
      But If they wouldn't have the ok from Microsoft there would be a chance that they start talking about copyright infringement.
- cptroot 3y ago
  As someone who's played probably too many hours of minecraft, these videos are nauseating. The way that all of the individual pieces exist, but have no consistency is terrifying. Random FoV changes, switching apparent texture packs, raytracing on or off, it's all still switching back and forth from moment to moment. These videos honestly give me less confidence in the approach, simply because I don't know that the model will be able to distinguish these world model parameters as "unchanging".
  - bongodongobob 3y ago
nopinsight 3y ago
AlphaGo and AlphaZero were able to achieve superhuman performance due to the availability of perfect simulators for the game of Go. There is no such simulator for the real world we live in (although pure LLMs sort of learn a rough, abstract representation of the world as perceived by humans.) Sora is an attempt to build such a simulator using deep learning. “Our results suggest that scaling video generation models is a promising path towards building general purpose simulators of the physical world.” General, superhuman robotic capabilities on the software side can be achieved once such a simulator is good enough. (Whether that can be achieved with this approach is still not certain.) Why superhuman? Larger context length than our working memory is an obvious one, but there will likely be other advantages such as using alternative sensory modalities and more granular simulation of details unfamiliar to most humans.
- roenxi 3y ago
  There is a perfect simulator of the real world available. It can be recorded with a camera! Once the researchers have a bit of time to get their bearings and figure out how to train an order of magnitude faster we'll get there.
  - throwaway290 3y ago
    That's still not a simulation if camera recording shows only what we see.
- Nathanba 3y ago
  Really interesting how this goes against my intuition. I would have imagined that it's infinitely easier to analyze a camera stream of the real world, then generate a polygonal representation of what you see (like you would do for a videogame) and then make AI decisions for that geometry. Instead the way that AI is going they rather skip it all and work directly on pixel data. Understanding of 3d geometry, perspective and physics is expected to evolve naturally from the training data.
  - stravant 3y ago
RayVR 3y ago
If there's one thing I've always wanted, it's shitty video knockoffs of real life. Can't wait to stream some AI hallucinations.
- danielbln 3y ago
  As if what you consume normally is actual real life. With blue screens, VFX, etc. you are already watching knockoffs of real life, and the shitty will become indistinguishable from reality before long.
- andybak 3y ago
  So is the only thing you watch on TV, unedited documentary footage? Also - ironic choice of username considering this comment!
sjwhevvvvvsj 3y ago
This is insanely good but look at the legs around 16 seconds in, they kinda morph through each other. Generally the legs are slightly unnerving. Still, god damn.
- tomxor 3y ago
  Yeah I saw that too, it does it in one other place, the feet are also sort of gliding around. If you look at the people in the background a lot of them are doing the same, and there are other temporal-mechanical inconsistencies like joints inverting half way through a movement, i guess due to it operating in 2D so when things are foreshortened from the camera angle they have the opportunity to suddenly jump into the wrong position, like twitchy inverse kinematics. Everything also has a sort of mushy feel about it, like nothing is anchored down, everything is swimming. Impressive all the same, but maybe these methods need to be paired with some old fashioned 3d rendering to serve as physically based guideline.
vunderba 3y ago
The improvement to temporal consistency given that the length of these generated videos is 3 to 4 times longer than anything else on the market (runway, pika, etc) is truly remarkable.
SushiHippie 3y ago
I like that this one shows some "fails", and not just the top of the top results: For example, the surfer is surfing in the air at the end: https://cdn.openai.com/tmp/s/prompting_7.mp4 https://cdn.openai.com/tmp/s/prompting_7.mp4 Or this "breaking" glass that does not break, but spills liquid in some weird way: https://cdn.openai.com/tmp/s/discussion_0.mp4 https://cdn.openai.com/tmp/s/discussion_0.mp4 Or the way this person walks: https://cdn.openai.com/tmp/s/a-woman-wearing-a-green-dress-and-a-sun-hat-taking-a-pleasant-stroll-in-Antarctica-during-a-winter-storm.mp4 https://cdn.openai.com/tmp/s/a-woman-wearing-a-green-dress-a... Or wherever this map is coming from: https://cdn.openai.com/tmp/s/a-woman-wearing-purple-overalls-and-cowboy-boots-taking-a-pleasant-stroll-in-Mumbai-India-during-a-beautiful-sunset.mp4 https://cdn.openai.com/tmp/s/a-woman-wearing-purple-overalls...
- coffeebeqn 3y ago
  The hyper realistic and plausible movement of the glass breaking makes this bizarrely fascinating. And it doesn’t give me the feeling of disgust the motion in the more primitive AI models did
  - deleted 3y ago
    [deleted]
- sega_sai 3y ago
  That is creepy...
  - SiempreViernes 3y ago
    Take a look at these adorable kangaroos to relax: https://cdn.openai.com/tmp/s/an-adorable-kangaroo-wearing-blue-jeans-and-a-white-t-shirt-taking-a-pleasant-stroll-in-Mumbai-India-during-a-beautiful-sunset.mp4 https://cdn.openai.com/tmp/s/an-adorable-kangaroo-wearing-bl... https://cdn.openai.com/tmp/s/an-adorable-kangaroo-wearing-a-green-dress-and-a-sun-hat-taking-a-pleasant-stroll-in-Mumbai-India-during-a-winter-storm.mp4 https://cdn.openai.com/tmp/s/an-adorable-kangaroo-wearing-a-...
liuliu 3y ago
Wow, this is really just scale-up DiT. We are going to see tons of similar models very soon.
empath-nirvana 3y ago
I think people might be missing what this enables. It can make plausible continuations of video, with realistic physics. What happens if this gets fast enough to work _in real time_. Connect this to a robot that has a real time camera feed. Have it constantly generate potential future continuations of the feed that it's getting -- maybe more than one. You have an autonomous robot building a real time model of the world around it and predicting the future. Give it some error correction based on well each prediction models the actual outcome and I think you're _really_ close to AGI. You can probably already imagine different ways to wire the output to text generation and controlling its own motions, etc, and predicting outcomes based on actions it, itself could plausibly take, and choosing the best one. It doesn't actually have to generate realistic imagery or imagery that doesn't have any mistakes or imagery that's high definition to be used in that way. How realistic is our own imagination of the world? Edit: I'm going to add a specific case. Imagine a house cleaning robot. It starts with an image of your living room. Then it creates a image of your living room after it's been cleaned. Then it interpolates a video _imagining itself cleaning the room_, then acts as much as it can to mimic what's in the video, then generates a new continuation, then acts, and so on. Imagine doing that several times a second, if necessary.
- aurareturn 3y ago
  Sounds like simulation theory is closer and closer to being proven.
  - littlestymaar 3y ago
    Our ability to build somewhat convincing simulations of thing has never been a proof of living in a simulation…
    - empath-nirvana 3y ago
      i mean everyone's mind builds a convincing internal simulation of reality and it's so good that most people think they're directly experiencing reality.
pedrovhb 3y ago
That's an interesting idea. Analogous to how LLMs are simply "text predictors" but end up having to learn a model of language and the world to correctly predict cohesive text, it makes sense that "video predictors" also have to learn a model of the world that makes sense. I wonder how many orders of magnitude further they have to evolve to be similarly useful.
myth_drannon 3y ago
Should I short all the 3d tool s/movies/vfx companies?
- drcode 3y ago
  I think not in the short term- I'm guessing the next step will be to use traditional tools to make a "draft" of a desired video, then "finish it off" with this kind of deep learning tech. So this tech will increase interest in existing 3d tools in the short term
- hackerlight 3y ago
  People working in vfx are incredibly gloomy today, they see the writing on the wall now, whether it's 1 or 5 years. There will still be a demand for human-created stuff but many of the jobs in advertising and stock footage will disappear.
- nojs 3y ago
  Arguably you should go long since once they integrate this into their products (as Adobe is doing) they have the distribution in place to monetise it, industry knowledge to combine it with existing workflows, etc.
guybedo 3y ago
I think it's Ylecun who stated a few times that video was the better way to train large models as it's more information dense. The results really are impressive. Being able to generate such high quality videos, to extend videos in the past and in the future shows how much the model "understands" the real world, objects interaction, 3D composition, etc... Although image generation already requires the model to know a lot about the world, i think there's really a huge gap with video generation where the model needs to "know" 3D, objects movements and interactions.
GaggiX 3y ago
The Minecraft demo makes me think that soon will be playing games directly from the output of one of these models, unlimited content.
- Jordan-117 3y ago
  Imagine something like GAN Theft Auto V powered by this technology: https://www.youtube.com/watch?v=udPY5rQVoW0 https://www.youtube.com/watch?v=udPY5rQVoW0
- kevingadd 3y ago
  While it seems plausible that eventually you could build a game around one of these models, the lack of an underlying state representation that you can permute in a precise way is a pretty strong barrier to anything resembling real user-and-system interaction. Even expressing pong through text prompts in a way that would produce desirable results in this is a tough challenge. I could imagine a text adventure game with a 'visual' component perhaps working if you got the model to maintain enough consistency in spaces and character appearances.
jk_tech 3y ago
This is some incredible and fascinating work! The applications seem endless. 1. High quality video or image from text 2. Taking in any content as input and generating forwards/backwards in time 3. Style transformation 4. Digital World simulation!
lbrito 3y ago
Okay, The Matrix can't be too far away now.
- mr_toad 3y ago
  Having the same cat enter the shot twice would definitely be a glitch in the Matrix.
- shiroiushi 3y ago
  Maybe before long they can make whole movies just with prompts giving a plot outline. Then they can finally make a sequel to The Matrix.
  - 93po 3y ago
    Finding Nemo except they're all in BDSM outfits
danavar 3y ago
While the Sora videos are impressive, are these really world simulators? While some notion of real-world physics probably exists somewhere within the model, doesn’t all the completely artificial training data corrupt it? Reasoning, logic, formal systems, and physics exist in a seemingly completely different, mathematical space than pure video. This is just a contrived, interesting viewpoint of the technology, right?
- Vecr 3y ago
  > Reasoning, logic, formal systems, and physics exist in a seemingly completely different, mathematical space than pure video. That's not true, AI systems in general have pretty strong mathematical proofs going back decades on what they can theoretically do, the problem is compute and general feasibility. AIXItl in theory would be able to learn reasoning, logic, formal systems, physics, human emotions, and a great deal of everything else just from watching videos. They would have to be videos of varied and useful things, but even if they were not, you'd at least get basic reasoning, logic, and physics.
- mr_toad 3y ago
  “it does not accurately model the physics of many basic interactions, like glass shattering.”
iliane5 3y ago
Watching an entirely generated video of someone painting is crazy. I can't wait to play with this but I can't even imagine how expensive it must be. They're training in full resolution and can generate up to a minute of video. Seeing how bad video generation was, I expected it would take a few more years to get to this but it seems like this is another case of "Add data & compute"(TM) where transformers prove once again they'll learn everything and be great at it
modeless 3y ago
> Other interactions, like eating food, do not always yield correct changes in object state So this is why they haven't shown Will Smith eating spaghetti. > These capabilities suggest that continued scaling of video models is a promising path towards the development of highly-capable simulators of the physical and digital world This is exciting for robotics. But an even closer application would be filling holes in gaussian splatting scenes. If you want to make a 3D walkthrough of a space you need to take hundreds to thousands of photos with seamless coverage of every possible angle, and you're still guaranteed to miss some. Seems like a model this capable could easily produce plausible reconstructions of hidden corners or close up detail or other things that would just be holes or blurry parts in a standard reconstruction. You might only need five or ten regular photos of a place to get a completely seamless and realistic 3D scene that you could explore from any angle. You could also do things like subtract people or other unwanted objects from the scene. Such an extrapolated reconstruction might not be completely faithful to reality in every detail, but I think this could enable lots of applications regardless.
- deleted 3y ago
  [deleted]
- SiempreViernes 3y ago
  Do note that "reconstruction" is not the right word, the proper characterisation of that sort of imputation is "artist impression": good for situations where the precise details doesn't matter. Though of course if the details doesn't matter maybe blurry is fine.
  - YeGoblynQueenne 3y ago
    Well, yeah, if the details don't matter then you don't need "highly-capable simulators of the physical and digital world". And if the details do matter, then good luck having a good enough simulation of the real world that you can invoke in real time in any kind of mobile hardware.
binary132 3y ago
Maybe this says more about me than about the technology, but I found the consistency of the Minecraft simulation super impressive.
- lanternfish 3y ago
  I was wondering how feasible it would be to make a Minecraft agent that had a running feed of the past few seconds, continued it off w/ SORA, fed the continuation into a (relatively) simple policy translator that just pulled out what the video showed as player inputs, and the inputted that. Presumably, this would work for non-minecraft applications, but Minecraft has a really standardized interface layer
  - binary132 3y ago
    It’s interesting to think about what the internal model of a NN of a game might be like, and whether it could be exposed with an API that someone could program against.
dang 3y ago
Related ongoing thread: Sora: Creating video from text - https://news.ycombinator.com/item?id=39386156 https://news.ycombinator.com/item?id=39386156 - Feb 2024 (1430 comments)
blueprint 3y ago
> Our results suggest that scaling video generation models is a promising path towards building general purpose simulators of the physical world. so they're gonna include the never-before-observed-but-predicted Unruh effect, as well? and other quantum theory? cool.. > For example, it does not accurately model the physics of many basic interactions, like glass shattering. ... oh Isn't all of the training predicated on visible, gathered data - rather than theory? if so, I don't think it's right to call these things simulators of the physical world if they don't include physical theory. DFT at least has some roots in theory.
- FeepingCreature 3y ago
  The point isn't "physical simulator" like supercomputers, it's "physical simulator" like the human brain.
  - blueprint 3y ago
    yeah, well maybe they should've said that, and they obviously would have if they intended to distinguish between the two of them instead of over inflating their claims in order to raise more money The simple counter argument is that they're claiming that there is potential for general purpose physical simulation. So please don't play with words. I thought we were supposed to be talking about something concrete instead of whatever you wanted to twist things into to suit your purpose either that, or they're not qualified to distinguish between the two options you presented, which is obviously not true right? the article's bazillion references have nothing to do with physics nor is there any substance behind the claim of physical simulation what is actually being simulated is synthesis of any video it's not that complicated and you and they must know it too it's double speak easily seen from miles away scammers have to downvote
pellucide 3y ago
I am a newbie to this area. Honest questions: Is this generating videos as streaming content e.g. like a mp4 video. As far as I can see, it is doing that. Is it possible for AI to actually produce the 3d models? What kind of compute resources are required to produce the 3d models.
- andybak 3y ago
  You can feed the output to NeRF or Gaussian Splat generators to produce 3d models: https://twitter.com/BenMildenhall/status/1758224827788468722 https://twitter.com/BenMildenhall/status/1758224827788468722 https://twitter.com/ScottieFoxTTV/status/1758272455603327455 https://twitter.com/ScottieFoxTTV/status/1758272455603327455 The key is that the video has spatial consistency. Once you've got that, then other existing tech can take the output and infer actual spatial forms.
proc0 3y ago
I don't know if there is research into this, didn't see it mentioned here, but this is the most probable path to something like AI consciousness and AGI. Of course it's highly speculative but video to world simulation is how the brain evolved and probably what is needed to have a robot behave like a living being. It would just do this in reverse, video input to inner world model, and use that for reasoning about the world. Extremely fascinating, and also scary this is happening so quickly.
zone411 3y ago
Video will be especially important for language models to grasp physical actions that are instinctive and obvious to humans but not explicitly detailed in text or video captions. I mentioned this in 2022: https://twitter.com/LechMazur/status/1607929403421462528 https://twitter.com/LechMazur/status/1607929403421462528 https://twitter.com/LechMazur/status/1619032477951213568 https://twitter.com/LechMazur/status/1619032477951213568
neurostimulant 3y ago
Where's the training data come from? Youtube?
advael 3y ago
People are obviously already pointing out the errors in various physical interactions shown in the demo videos, including the research team themselves, and I think the plausiblity of the generated videos will likely improve as they work on the model more. However, I think the major reason this generation -> simulation leap might be harder leap than they think is actually a plausibility/accuracy distinction. Generative models are general and versatile compared to predictive models, but they're intrinsically learning an objective that assesses its extrapolations on spatial or sequential (or in the case of video, both) plausibility, which has a lot more degrees of freedom than accuracy. In other words, the ability to create reasonable-enough hypotheses for what the next frame or the next pixel over could end up not being enough. The optimistic scenario is that it's possible to get to a simulation by narrowing this hypothesis-space enough to accurately model reality. In other words, it's possible that this is just something that could fall out of the plausibility being continuously improved, like the subset of plausible hypotheses shrinks as the model gets better, and eventually we get a reality-predictor, but I think there are good reasons to think that's far from guaranteed. I'd be curious to see what happens if you restrict training data to unaltered camera footage rather than allowing anything fictitious, but the least optimistic possibility is that this kind of capability is necessary but not sufficient for adequate prediction (or slightly more optimistically, can only do so with amounts of resolution that are currently infeasible, or something). Some of the reasons the less optimistic scenarios seem likely is that the kinds of extrapolation errors this model makes are of similar character to those of LLMs: extrapolation follows a gradient of smooth apparent transitions rather than some underlying logic about the objects portrayed, and sometimes seems to just sort of ignore situations that are far enough outside of what it's seen rather than reconcile them. For example, the tidal wave/historical hall example is a scenario unlikely to have been in the training data. Sure, there's the funny bit at the end where the surfer appears to levitate in the air, but there's a much larger issue with how these two contrasting scenes interact, or rather fail to. What we see looks a lot more like a scene of surfing superimposed via photoshop or something on a still image of the hall, as there's no evidence of the water interacting with the seats or walls in the hall at all. The model will just roll with whatever you tell it to do as best it can, but it's not doing something like modeling "what would happen if" that implausible scenario played out, and even doing it poorly would be a better sign for this doing something like "simulating" the described scenario. Instead, we have impressive results for prompts that likely strongly correspond to scenes the model may have seen, and evidence of a lack of composition in cases where a particular composition is unlikely to have been seen and needs some underlying understanding of how it "would" work that is visible to us
pmontra 3y ago
The video with the two MTBs going downhill: it seems to me that the long left turn that begins a few second into the video is way too long. It's easy to misjudge that kind of things (try to draw a road race track by looking at a single lap of it) but it could end up below the point where it started, or too close to it to be physically realistic. I was expecting to see a right turn at any moment but it kept going left. It could be another consequence of the lack of real knowledge about the world, similar to the glass shattering example at the end of the article.
anonyfox 3y ago
If they would allow this (maybe a premium+ model) they could soon destroy the whole porn industry. not the websites, but the (often abused) sex workers. Everyone could describe that fetish they are into and get it visualized instantly without the need of physical human suffering to produce these videos. I know its a delicate topic people (especially in the US) don't want to speak about at all, but damn, this is a giant market and could do humanity good if done well.
- michalf6 3y ago
  There are thousands of porn consumers with destroyed reward circuitry per every porn actor, of which few are mistreated and the majority are compensated very well. Producing a neverending supply of wireheading-like addictive stimuli is the farthest possible thing from a good for humanity. Want to do good in this area - work on ways to limit consumption.
  - 93po 3y ago
    This is easily proven inaccurate. The largest network of sex workers is OnlyFans. Go look up stats for OF creators. It's 99.5% people making basically zero money, and 0.01% making 99% of the money.
- kenning 3y ago
  the reaction of a pedo or worse lol
koonsolo 3y ago
Yesterday I was watching a movie on Netflix and thought to myself, what if Netflix generated a movie based on what I want to see and what I like. Plus, it could generate it in real time and take my responses into account. I look bored? Spice it up, etc. Today such a thing seems closer than I thought.
tokai 3y ago
Ugh, AI generated images everywhere is already annoying enough. Now we're gonna have these factitious videos clogging up everything, and I'll have to explain my old neighbor that Biden did infact not eat a fetus again and again.
- SuaveSteve 3y ago
  100%. It's actually gotten even more dull once they started fixing the fingers. But it's too much; you start to realise that it's just so uninspired. Maybe what this will ultimately do is allow good writers to bring their ideas to life (I hope).
bawana 3y ago
SORA.....the entire movie industry is now out of a job.
- stephenitis 3y ago
  Did you actually read the post? This is not the case, yet. Also the concept of learning to simulating the world seems more important than just the media and content implications.
yakito 3y ago
Does anyone know why most of the videos are in slow motion?
- 93po 3y ago
  I'm guessing that either the shots its trained on have a propensity to be slo-mo (like puppies playing in a field) or making it slow-motion makes unnatural movement a lot less obvious
chankstein38 3y ago
This is the second Sora announcement I've seen. Am I missing how I can play with it? The examples in the papers are all well and good but I want to get my hands on it and try it.
anirudhv27 3y ago
What makes OpenAI so far ahead of all of these other research firms (or even startups like Pika, Runway, etc.)? I feel like I see so many examples of fields where progress is being made all across and OpenAI suddenly swoops in with an insane breakthrough lightyears ahead of everyone else.
htrp 3y ago
> We empirically find that training on videos at their native aspect ratios improves composition and framing. We compare Sora against a version of our model that crops all training videos to be square, which is common practice when training generative models. The model trained on square crops (left) sometimes generates videos where the subject is only partially in view. In comparison, videos from Sora (right)s have improved framing. Every cv preprocessing pipeline is in shambles now.
newswasboring 3y ago
This is a totally silly thought, but I still want to get it out there. > Other interactions, like eating food, do not always yield correct changes in object state Can this be because we just don't shoot a lot of people eating? I think it is general advice to not show people eating on camera for various reasons. I wonder if we know if that kind of topic bias exists in the dataset.
exe34 3y ago
The current development of AI seems like speed run of Crystal Society in terms of their interaction with the world. The only thing missing is the Inner Purpose.

361 points by linksbro 3y ago

I find it wild that this model does not have explicit 3D prior, yet learns to generate videos with such 3D consistency, you can directly train a 3D representation (NeRF-like) from those videos: https://twitter.com/BenMildenhall/status/1758224827788468722 https://twitter.com/BenMildenhall/status/1758224827788468722

nodja 3y ago
This is also true for 2D diffusion models[1]. I suppose they need to understand how 3d works for stuff like lighting/shadows/object occlusion, etc. [1] https://dreamfusion3d.github.io/ https://dreamfusion3d.github.io/
crooked-v 3y ago
That leaves me wondering if it'd be possible to get some variant of the model to directly output 3D meshes and camera animation instead of an image.
Nihilartikel 3y ago
I was similarly astonished at this adaptation of stable diffusion to make HDR spherical environment maps from existing images- https://diffusionlight.github.io/ https://diffusionlight.github.io/ The crazy thing is that they do it by prompting the model to in paint a chrome sphere into the middle of the image to reflect what is behind the camera! The model can interpret the context and dream up what is plausibly in the whole environment.
- emadm 3y ago
  Yeah we were surprised by that, video models are great 3d prices and image models are great video model priors
TOMDM 3y ago
I wonder how much it'd improve if trained on stereo image data.
- QuadmasterXLII 3y ago
  Moving camera is just stereo.
larschdk 3y ago
You aren't looking carefully enough. I find so many inconsistencies in these examples. Perspectives that are completely wrong when the camera rotates. Windows that shift perspective, patios that are suddenly deep/shallow. Shadows that appear/disappear as the camera shifts. In other examples; paths, objects, people suddenly appearing or disappearing out of nowhere. A stone turning into a person. A horse that suddenly has a second head, then becomes a separate horse with only two legs. It is impressive at a glance, but if you pay attention, it is more like dreaming than realism (images conjured out of other images, without attention to long term temporal, spatial, and causal consistency). I'm hardly more impressed that Google's deep dream, which is 10 years old.
- beezlebroxxxxxx 3y ago
  Yeah, it seems to have a hard time processing lens distortion in particular which gives a very weird quality. It's actually bending things, or trying to fill in the gaps, instead of distorting the image in the "correct" way.
- lairv 3y ago
  You can literally run 3D algorithms like NeRF or COLMAP on those videos (check the tweet I sent), it's not my opinion, those videos are sufficiently 3D consistent that you can extract 3D geometry from them Surely it's not perfect, but this was not the case for previous video generation algorithms

data-ottawa 3y ago

I know the main post has been getting a lot of reaction, but this page absolutely blew me away. The results are striking. The robot examples are very underwhelming, but the people and background people are all very well done, and at a level much better than most static image diffusion models produce. Generating the same people as the interact with objects is also not something I expected a model like this to do well so soon.

colesantiago 3y ago

Damn, even minecraft videos being simulated, this is crazy to see from OpenAI. Edit, changed the links to the direct ones! https://cdn.openai.com/tmp/s/simulation_6.mp4 https://cdn.openai.com/tmp/s/simulation_6.mp4 https://cdn.openai.com/tmp/s/simulation_7.mp4 https://cdn.openai.com/tmp/s/simulation_7.mp4

SushiHippie 3y ago
But it starts to make sense, when you think about the fact that Minecraft is owned by Microsoft. example video links from TFA: https://cdn.openai.com/tmp/s/simulation_6.mp4 https://cdn.openai.com/tmp/s/simulation_6.mp4 https://cdn.openai.com/tmp/s/simulation_7.mp4 https://cdn.openai.com/tmp/s/simulation_7.mp4
- empath-nirvana 3y ago
  I don't think that has anything to do with it. There's just millions of hours of minecraft video on youtube and twitch.
  - SushiHippie 3y ago
    But If they wouldn't have the ok from Microsoft there would be a chance that they start talking about copyright infringement.
cptroot 3y ago
As someone who's played probably too many hours of minecraft, these videos are nauseating. The way that all of the individual pieces exist, but have no consistency is terrifying. Random FoV changes, switching apparent texture packs, raytracing on or off, it's all still switching back and forth from moment to moment. These videos honestly give me less confidence in the approach, simply because I don't know that the model will be able to distinguish these world model parameters as "unchanging".
- bongodongobob 3y ago

nopinsight 3y ago

AlphaGo and AlphaZero were able to achieve superhuman performance due to the availability of perfect simulators for the game of Go. There is no such simulator for the real world we live in (although pure LLMs sort of learn a rough, abstract representation of the world as perceived by humans.) Sora is an attempt to build such a simulator using deep learning. “Our results suggest that scaling video generation models is a promising path towards building general purpose simulators of the physical world.” General, superhuman robotic capabilities on the software side can be achieved once such a simulator is good enough. (Whether that can be achieved with this approach is still not certain.) Why superhuman? Larger context length than our working memory is an obvious one, but there will likely be other advantages such as using alternative sensory modalities and more granular simulation of details unfamiliar to most humans.

roenxi 3y ago
There is a perfect simulator of the real world available. It can be recorded with a camera! Once the researchers have a bit of time to get their bearings and figure out how to train an order of magnitude faster we'll get there.
- throwaway290 3y ago
  That's still not a simulation if camera recording shows only what we see.
Nathanba 3y ago
Really interesting how this goes against my intuition. I would have imagined that it's infinitely easier to analyze a camera stream of the real world, then generate a polygonal representation of what you see (like you would do for a videogame) and then make AI decisions for that geometry. Instead the way that AI is going they rather skip it all and work directly on pixel data. Understanding of 3d geometry, perspective and physics is expected to evolve naturally from the training data.
- stravant 3y ago

RayVR 3y ago

If there's one thing I've always wanted, it's shitty video knockoffs of real life. Can't wait to stream some AI hallucinations.

danielbln 3y ago
As if what you consume normally is actual real life. With blue screens, VFX, etc. you are already watching knockoffs of real life, and the shitty will become indistinguishable from reality before long.
andybak 3y ago
So is the only thing you watch on TV, unedited documentary footage? Also - ironic choice of username considering this comment!

sjwhevvvvvsj 3y ago

This is insanely good but look at the legs around 16 seconds in, they kinda morph through each other. Generally the legs are slightly unnerving. Still, god damn.

tomxor 3y ago
Yeah I saw that too, it does it in one other place, the feet are also sort of gliding around. If you look at the people in the background a lot of them are doing the same, and there are other temporal-mechanical inconsistencies like joints inverting half way through a movement, i guess due to it operating in 2D so when things are foreshortened from the camera angle they have the opportunity to suddenly jump into the wrong position, like twitchy inverse kinematics. Everything also has a sort of mushy feel about it, like nothing is anchored down, everything is swimming. Impressive all the same, but maybe these methods need to be paired with some old fashioned 3d rendering to serve as physically based guideline.

vunderba 3y ago

The improvement to temporal consistency given that the length of these generated videos is 3 to 4 times longer than anything else on the market (runway, pika, etc) is truly remarkable.

SushiHippie 3y ago

I like that this one shows some "fails", and not just the top of the top results: For example, the surfer is surfing in the air at the end: https://cdn.openai.com/tmp/s/prompting_7.mp4 https://cdn.openai.com/tmp/s/prompting_7.mp4 Or this "breaking" glass that does not break, but spills liquid in some weird way: https://cdn.openai.com/tmp/s/discussion_0.mp4 https://cdn.openai.com/tmp/s/discussion_0.mp4 Or the way this person walks: https://cdn.openai.com/tmp/s/a-woman-wearing-a-green-dress-and-a-sun-hat-taking-a-pleasant-stroll-in-Antarctica-during-a-winter-storm.mp4 https://cdn.openai.com/tmp/s/a-woman-wearing-a-green-dress-a... Or wherever this map is coming from: https://cdn.openai.com/tmp/s/a-woman-wearing-purple-overalls-and-cowboy-boots-taking-a-pleasant-stroll-in-Mumbai-India-during-a-beautiful-sunset.mp4 https://cdn.openai.com/tmp/s/a-woman-wearing-purple-overalls...

coffeebeqn 3y ago
The hyper realistic and plausible movement of the glass breaking makes this bizarrely fascinating. And it doesn’t give me the feeling of disgust the motion in the more primitive AI models did
- deleted 3y ago
  [deleted]
sega_sai 3y ago
That is creepy...
- SiempreViernes 3y ago
  Take a look at these adorable kangaroos to relax: https://cdn.openai.com/tmp/s/an-adorable-kangaroo-wearing-blue-jeans-and-a-white-t-shirt-taking-a-pleasant-stroll-in-Mumbai-India-during-a-beautiful-sunset.mp4 https://cdn.openai.com/tmp/s/an-adorable-kangaroo-wearing-bl... https://cdn.openai.com/tmp/s/an-adorable-kangaroo-wearing-a-green-dress-and-a-sun-hat-taking-a-pleasant-stroll-in-Mumbai-India-during-a-winter-storm.mp4 https://cdn.openai.com/tmp/s/an-adorable-kangaroo-wearing-a-...

liuliu 3y ago

Wow, this is really just scale-up DiT. We are going to see tons of similar models very soon.

empath-nirvana 3y ago

I think people might be missing what this enables. It can make plausible continuations of video, with realistic physics. What happens if this gets fast enough to work _in real time_. Connect this to a robot that has a real time camera feed. Have it constantly generate potential future continuations of the feed that it's getting -- maybe more than one. You have an autonomous robot building a real time model of the world around it and predicting the future. Give it some error correction based on well each prediction models the actual outcome and I think you're _really_ close to AGI. You can probably already imagine different ways to wire the output to text generation and controlling its own motions, etc, and predicting outcomes based on actions it, itself could plausibly take, and choosing the best one. It doesn't actually have to generate realistic imagery or imagery that doesn't have any mistakes or imagery that's high definition to be used in that way. How realistic is our own imagination of the world? Edit: I'm going to add a specific case. Imagine a house cleaning robot. It starts with an image of your living room. Then it creates a image of your living room after it's been cleaned. Then it interpolates a video _imagining itself cleaning the room_, then acts as much as it can to mimic what's in the video, then generates a new continuation, then acts, and so on. Imagine doing that several times a second, if necessary.

aurareturn 3y ago
Sounds like simulation theory is closer and closer to being proven.
- littlestymaar 3y ago
  Our ability to build somewhat convincing simulations of thing has never been a proof of living in a simulation…
  - empath-nirvana 3y ago
    i mean everyone's mind builds a convincing internal simulation of reality and it's so good that most people think they're directly experiencing reality.

pedrovhb 3y ago

That's an interesting idea. Analogous to how LLMs are simply "text predictors" but end up having to learn a model of language and the world to correctly predict cohesive text, it makes sense that "video predictors" also have to learn a model of the world that makes sense. I wonder how many orders of magnitude further they have to evolve to be similarly useful.

myth_drannon 3y ago

Should I short all the 3d tool s/movies/vfx companies?

drcode 3y ago
I think not in the short term- I'm guessing the next step will be to use traditional tools to make a "draft" of a desired video, then "finish it off" with this kind of deep learning tech. So this tech will increase interest in existing 3d tools in the short term
hackerlight 3y ago
People working in vfx are incredibly gloomy today, they see the writing on the wall now, whether it's 1 or 5 years. There will still be a demand for human-created stuff but many of the jobs in advertising and stock footage will disappear.
nojs 3y ago
Arguably you should go long since once they integrate this into their products (as Adobe is doing) they have the distribution in place to monetise it, industry knowledge to combine it with existing workflows, etc.

guybedo 3y ago

I think it's Ylecun who stated a few times that video was the better way to train large models as it's more information dense. The results really are impressive. Being able to generate such high quality videos, to extend videos in the past and in the future shows how much the model "understands" the real world, objects interaction, 3D composition, etc... Although image generation already requires the model to know a lot about the world, i think there's really a huge gap with video generation where the model needs to "know" 3D, objects movements and interactions.

GaggiX 3y ago

The Minecraft demo makes me think that soon will be playing games directly from the output of one of these models, unlimited content.

Jordan-117 3y ago
Imagine something like GAN Theft Auto V powered by this technology: https://www.youtube.com/watch?v=udPY5rQVoW0 https://www.youtube.com/watch?v=udPY5rQVoW0
kevingadd 3y ago
While it seems plausible that eventually you could build a game around one of these models, the lack of an underlying state representation that you can permute in a precise way is a pretty strong barrier to anything resembling real user-and-system interaction. Even expressing pong through text prompts in a way that would produce desirable results in this is a tough challenge. I could imagine a text adventure game with a 'visual' component perhaps working if you got the model to maintain enough consistency in spaces and character appearances.

jk_tech 3y ago

This is some incredible and fascinating work! The applications seem endless. 1. High quality video or image from text 2. Taking in any content as input and generating forwards/backwards in time 3. Style transformation 4. Digital World simulation!

lbrito 3y ago

Okay, The Matrix can't be too far away now.

mr_toad 3y ago
Having the same cat enter the shot twice would definitely be a glitch in the Matrix.
shiroiushi 3y ago
Maybe before long they can make whole movies just with prompts giving a plot outline. Then they can finally make a sequel to The Matrix.
- 93po 3y ago
  Finding Nemo except they're all in BDSM outfits

danavar 3y ago

While the Sora videos are impressive, are these really world simulators? While some notion of real-world physics probably exists somewhere within the model, doesn’t all the completely artificial training data corrupt it? Reasoning, logic, formal systems, and physics exist in a seemingly completely different, mathematical space than pure video. This is just a contrived, interesting viewpoint of the technology, right?

Vecr 3y ago
> Reasoning, logic, formal systems, and physics exist in a seemingly completely different, mathematical space than pure video. That's not true, AI systems in general have pretty strong mathematical proofs going back decades on what they can theoretically do, the problem is compute and general feasibility. AIXItl in theory would be able to learn reasoning, logic, formal systems, physics, human emotions, and a great deal of everything else just from watching videos. They would have to be videos of varied and useful things, but even if they were not, you'd at least get basic reasoning, logic, and physics.
mr_toad 3y ago
“it does not accurately model the physics of many basic interactions, like glass shattering.”

iliane5 3y ago

Watching an entirely generated video of someone painting is crazy. I can't wait to play with this but I can't even imagine how expensive it must be. They're training in full resolution and can generate up to a minute of video. Seeing how bad video generation was, I expected it would take a few more years to get to this but it seems like this is another case of "Add data & compute"(TM) where transformers prove once again they'll learn everything and be great at it

modeless 3y ago

> Other interactions, like eating food, do not always yield correct changes in object state So this is why they haven't shown Will Smith eating spaghetti. > These capabilities suggest that continued scaling of video models is a promising path towards the development of highly-capable simulators of the physical and digital world This is exciting for robotics. But an even closer application would be filling holes in gaussian splatting scenes. If you want to make a 3D walkthrough of a space you need to take hundreds to thousands of photos with seamless coverage of every possible angle, and you're still guaranteed to miss some. Seems like a model this capable could easily produce plausible reconstructions of hidden corners or close up detail or other things that would just be holes or blurry parts in a standard reconstruction. You might only need five or ten regular photos of a place to get a completely seamless and realistic 3D scene that you could explore from any angle. You could also do things like subtract people or other unwanted objects from the scene. Such an extrapolated reconstruction might not be completely faithful to reality in every detail, but I think this could enable lots of applications regardless.

deleted 3y ago
[deleted]
SiempreViernes 3y ago
Do note that "reconstruction" is not the right word, the proper characterisation of that sort of imputation is "artist impression": good for situations where the precise details doesn't matter. Though of course if the details doesn't matter maybe blurry is fine.
- YeGoblynQueenne 3y ago
  Well, yeah, if the details don't matter then you don't need "highly-capable simulators of the physical and digital world". And if the details do matter, then good luck having a good enough simulation of the real world that you can invoke in real time in any kind of mobile hardware.

binary132 3y ago

Maybe this says more about me than about the technology, but I found the consistency of the Minecraft simulation super impressive.

lanternfish 3y ago
I was wondering how feasible it would be to make a Minecraft agent that had a running feed of the past few seconds, continued it off w/ SORA, fed the continuation into a (relatively) simple policy translator that just pulled out what the video showed as player inputs, and the inputted that. Presumably, this would work for non-minecraft applications, but Minecraft has a really standardized interface layer
- binary132 3y ago
  It’s interesting to think about what the internal model of a NN of a game might be like, and whether it could be exposed with an API that someone could program against.

dang 3y ago

Related ongoing thread: Sora: Creating video from text - https://news.ycombinator.com/item?id=39386156 https://news.ycombinator.com/item?id=39386156 - Feb 2024 (1430 comments)

blueprint 3y ago

> Our results suggest that scaling video generation models is a promising path towards building general purpose simulators of the physical world. so they're gonna include the never-before-observed-but-predicted Unruh effect, as well? and other quantum theory? cool.. > For example, it does not accurately model the physics of many basic interactions, like glass shattering. ... oh Isn't all of the training predicated on visible, gathered data - rather than theory? if so, I don't think it's right to call these things simulators of the physical world if they don't include physical theory. DFT at least has some roots in theory.

FeepingCreature 3y ago
The point isn't "physical simulator" like supercomputers, it's "physical simulator" like the human brain.
- blueprint 3y ago
  yeah, well maybe they should've said that, and they obviously would have if they intended to distinguish between the two of them instead of over inflating their claims in order to raise more money The simple counter argument is that they're claiming that there is potential for general purpose physical simulation. So please don't play with words. I thought we were supposed to be talking about something concrete instead of whatever you wanted to twist things into to suit your purpose either that, or they're not qualified to distinguish between the two options you presented, which is obviously not true right? the article's bazillion references have nothing to do with physics nor is there any substance behind the claim of physical simulation what is actually being simulated is synthesis of any video it's not that complicated and you and they must know it too it's double speak easily seen from miles away scammers have to downvote

pellucide 3y ago

I am a newbie to this area. Honest questions: Is this generating videos as streaming content e.g. like a mp4 video. As far as I can see, it is doing that. Is it possible for AI to actually produce the 3d models? What kind of compute resources are required to produce the 3d models.

andybak 3y ago
You can feed the output to NeRF or Gaussian Splat generators to produce 3d models: https://twitter.com/BenMildenhall/status/1758224827788468722 https://twitter.com/BenMildenhall/status/1758224827788468722 https://twitter.com/ScottieFoxTTV/status/1758272455603327455 https://twitter.com/ScottieFoxTTV/status/1758272455603327455 The key is that the video has spatial consistency. Once you've got that, then other existing tech can take the output and infer actual spatial forms.

proc0 3y ago

I don't know if there is research into this, didn't see it mentioned here, but this is the most probable path to something like AI consciousness and AGI. Of course it's highly speculative but video to world simulation is how the brain evolved and probably what is needed to have a robot behave like a living being. It would just do this in reverse, video input to inner world model, and use that for reasoning about the world. Extremely fascinating, and also scary this is happening so quickly.

zone411 3y ago

Video will be especially important for language models to grasp physical actions that are instinctive and obvious to humans but not explicitly detailed in text or video captions. I mentioned this in 2022: https://twitter.com/LechMazur/status/1607929403421462528 https://twitter.com/LechMazur/status/1607929403421462528 https://twitter.com/LechMazur/status/1619032477951213568 https://twitter.com/LechMazur/status/1619032477951213568

neurostimulant 3y ago

Where's the training data come from? Youtube?

advael 3y ago

People are obviously already pointing out the errors in various physical interactions shown in the demo videos, including the research team themselves, and I think the plausiblity of the generated videos will likely improve as they work on the model more. However, I think the major reason this generation -> simulation leap might be harder leap than they think is actually a plausibility/accuracy distinction. Generative models are general and versatile compared to predictive models, but they're intrinsically learning an objective that assesses its extrapolations on spatial or sequential (or in the case of video, both) plausibility, which has a lot more degrees of freedom than accuracy. In other words, the ability to create reasonable-enough hypotheses for what the next frame or the next pixel over could end up not being enough. The optimistic scenario is that it's possible to get to a simulation by narrowing this hypothesis-space enough to accurately model reality. In other words, it's possible that this is just something that could fall out of the plausibility being continuously improved, like the subset of plausible hypotheses shrinks as the model gets better, and eventually we get a reality-predictor, but I think there are good reasons to think that's far from guaranteed. I'd be curious to see what happens if you restrict training data to unaltered camera footage rather than allowing anything fictitious, but the least optimistic possibility is that this kind of capability is necessary but not sufficient for adequate prediction (or slightly more optimistically, can only do so with amounts of resolution that are currently infeasible, or something). Some of the reasons the less optimistic scenarios seem likely is that the kinds of extrapolation errors this model makes are of similar character to those of LLMs: extrapolation follows a gradient of smooth apparent transitions rather than some underlying logic about the objects portrayed, and sometimes seems to just sort of ignore situations that are far enough outside of what it's seen rather than reconcile them. For example, the tidal wave/historical hall example is a scenario unlikely to have been in the training data. Sure, there's the funny bit at the end where the surfer appears to levitate in the air, but there's a much larger issue with how these two contrasting scenes interact, or rather fail to. What we see looks a lot more like a scene of surfing superimposed via photoshop or something on a still image of the hall, as there's no evidence of the water interacting with the seats or walls in the hall at all. The model will just roll with whatever you tell it to do as best it can, but it's not doing something like modeling "what would happen if" that implausible scenario played out, and even doing it poorly would be a better sign for this doing something like "simulating" the described scenario. Instead, we have impressive results for prompts that likely strongly correspond to scenes the model may have seen, and evidence of a lack of composition in cases where a particular composition is unlikely to have been seen and needs some underlying understanding of how it "would" work that is visible to us

pmontra 3y ago

The video with the two MTBs going downhill: it seems to me that the long left turn that begins a few second into the video is way too long. It's easy to misjudge that kind of things (try to draw a road race track by looking at a single lap of it) but it could end up below the point where it started, or too close to it to be physically realistic. I was expecting to see a right turn at any moment but it kept going left. It could be another consequence of the lack of real knowledge about the world, similar to the glass shattering example at the end of the article.

anonyfox 3y ago

If they would allow this (maybe a premium+ model) they could soon destroy the whole porn industry. not the websites, but the (often abused) sex workers. Everyone could describe that fetish they are into and get it visualized instantly without the need of physical human suffering to produce these videos. I know its a delicate topic people (especially in the US) don't want to speak about at all, but damn, this is a giant market and could do humanity good if done well.

michalf6 3y ago
There are thousands of porn consumers with destroyed reward circuitry per every porn actor, of which few are mistreated and the majority are compensated very well. Producing a neverending supply of wireheading-like addictive stimuli is the farthest possible thing from a good for humanity. Want to do good in this area - work on ways to limit consumption.
- 93po 3y ago
  This is easily proven inaccurate. The largest network of sex workers is OnlyFans. Go look up stats for OF creators. It's 99.5% people making basically zero money, and 0.01% making 99% of the money.
kenning 3y ago
the reaction of a pedo or worse lol

koonsolo 3y ago

Yesterday I was watching a movie on Netflix and thought to myself, what if Netflix generated a movie based on what I want to see and what I like. Plus, it could generate it in real time and take my responses into account. I look bored? Spice it up, etc. Today such a thing seems closer than I thought.

tokai 3y ago

Ugh, AI generated images everywhere is already annoying enough. Now we're gonna have these factitious videos clogging up everything, and I'll have to explain my old neighbor that Biden did infact not eat a fetus again and again.

SuaveSteve 3y ago
100%. It's actually gotten even more dull once they started fixing the fingers. But it's too much; you start to realise that it's just so uninspired. Maybe what this will ultimately do is allow good writers to bring their ideas to life (I hope).

bawana 3y ago

SORA.....the entire movie industry is now out of a job.

stephenitis 3y ago
Did you actually read the post? This is not the case, yet. Also the concept of learning to simulating the world seems more important than just the media and content implications.

yakito 3y ago

Does anyone know why most of the videos are in slow motion?

93po 3y ago
I'm guessing that either the shots its trained on have a propensity to be slo-mo (like puppies playing in a field) or making it slow-motion makes unnatural movement a lot less obvious

chankstein38 3y ago

This is the second Sora announcement I've seen. Am I missing how I can play with it? The examples in the papers are all well and good but I want to get my hands on it and try it.

anirudhv27 3y ago

What makes OpenAI so far ahead of all of these other research firms (or even startups like Pika, Runway, etc.)? I feel like I see so many examples of fields where progress is being made all across and OpenAI suddenly swoops in with an insane breakthrough lightyears ahead of everyone else.

htrp 3y ago

> We empirically find that training on videos at their native aspect ratios improves composition and framing. We compare Sora against a version of our model that crops all training videos to be square, which is common practice when training generative models. The model trained on square crops (left) sometimes generates videos where the subject is only partially in view. In comparison, videos from Sora (right)s have improved framing. Every cv preprocessing pipeline is in shambles now.

newswasboring 3y ago

This is a totally silly thought, but I still want to get it out there. > Other interactions, like eating food, do not always yield correct changes in object state Can this be because we just don't shoot a lot of people eating? I think it is general advice to not show people eating on camera for various reasons. I wonder if we know if that kind of topic bias exists in the dataset.

exe34 3y ago

The current development of AI seems like speed run of Crystal Society in terms of their interaction with the world. The only thing missing is the Inner Purpose.