2 ms·

I don't see this as something that would be hard to overcome. Sora for instance has already shown the ability for a diffusion model to maintain object permanenc

I don't see this as something that would be hard to overcome. Sora for instance has already shown the ability for a diffusion model to maintain object permanence. Flux recently too has shown the ability to render the same person in many different poses or images.

idunnoman1222 2y ago
Where does a sora video turn around backwards? I don’t even maintain such consistency in my dreams.
idunnoman1222 2y ago
Where does a sora video turn around backwards? I can’t maintain such consistency in my own dreams.
- Workaccount2 2y ago
  I don't know of an example (not to say it doesn't exist) but the problem is fundamentally the same as things moving out of sight/out of frame and coming back again.
  - Jensson 2y ago
    > the problem is fundamentally the same as things moving out of sight/out of frame and coming back again Maybe it is, but doing that with the entire scene instead of just a small part of it makes the problem massively harder, as the model needs to grow exponentially to remember more things. It isn't something that we will manage anytime soon, maybe 10-20 years with current architecture and same compute progress. Then you make that even harder by remembering a whole game level? No, ain't gonna happen in our lifetimes without massive changes to the architecture. They would need to make a different model keep track of level state etc, not just an image to image model.
    - Workaccount2 2y ago
      10 to 20 years sounds wildly pessimistic In this sora video the dragon covers half the scene, and its basically identical when it is revealed again ~5 seconds later, or about 150 frames later. The is lots of evidence (and some studies) that these models are in fact building internal world models. https://www.youtube.com/watch?v=LXJ-yLiktDU https://www.youtube.com/watch?v=LXJ-yLiktDU Buckle in, the train is moving way faster. I don't think there would be much surprise if this is solved in the next few generations of video generators. The first generation is already doing very well.
      - Jensson 2y ago
        Did you watch the video, it is completely different after the dragon goes past? Its still a flag there, but everything else changed. Even the stores in the background changed, the mass of people is completely different with no hint of anyone moving there etc. You always get this from AI enthusiast, they come and post "proof" that disproves their own point.
        HappMacDonald 2y ago
        I'm not GP, but running over that video I'm actually having a hard time finding any detail present before the dragon obscures them not either exit frame right when the camera pans left slightly near the end or not re-appear with reasonably crisp detail after the dragon gets out of the way. Most of the mob of people are indistinct, but there is a woman in a lime green coat who is visible, and then obstructed by the dragon twice (beard and ribbon) and reappears fine. Unfortunately when dragon fully moves past she has been lost to frame right. There is another person in black holding a red satchel which is visible both before and after the dragon has passed. Nothing about the storefronts appear to change. The complex sign full of Chinese text (which might be gibberish text: it's highly stylized and I don't know Chinese) appears to survive the dragon passing without even any changes to the individual ideograms. There is also a red box shaped like a Chinese paper lantern with a single gold ideogram on it at the store entrance which spends most of the video obscured by the dragon and is still in the same location after it passes (though video artifacting makes it more challenging to verify that that ideogram is unchanged it certainly does not appear substantially different) What detail are you seeing that is different before and after the obstruction?

idunnoman1222 2y ago

Where does a sora video turn around backwards? I don’t even maintain such consistency in my dreams.

idunnoman1222 2y ago

Where does a sora video turn around backwards? I can’t maintain such consistency in my own dreams.

Workaccount2 2y ago
I don't know of an example (not to say it doesn't exist) but the problem is fundamentally the same as things moving out of sight/out of frame and coming back again.
- Jensson 2y ago
  > the problem is fundamentally the same as things moving out of sight/out of frame and coming back again Maybe it is, but doing that with the entire scene instead of just a small part of it makes the problem massively harder, as the model needs to grow exponentially to remember more things. It isn't something that we will manage anytime soon, maybe 10-20 years with current architecture and same compute progress. Then you make that even harder by remembering a whole game level? No, ain't gonna happen in our lifetimes without massive changes to the architecture. They would need to make a different model keep track of level state etc, not just an image to image model.
  - Workaccount2 2y ago
    10 to 20 years sounds wildly pessimistic In this sora video the dragon covers half the scene, and its basically identical when it is revealed again ~5 seconds later, or about 150 frames later. The is lots of evidence (and some studies) that these models are in fact building internal world models. https://www.youtube.com/watch?v=LXJ-yLiktDU https://www.youtube.com/watch?v=LXJ-yLiktDU Buckle in, the train is moving way faster. I don't think there would be much surprise if this is solved in the next few generations of video generators. The first generation is already doing very well.
    - Jensson 2y ago
      Did you watch the video, it is completely different after the dragon goes past? Its still a flag there, but everything else changed. Even the stores in the background changed, the mass of people is completely different with no hint of anyone moving there etc. You always get this from AI enthusiast, they come and post "proof" that disproves their own point.
      - HappMacDonald 2y ago
        I'm not GP, but running over that video I'm actually having a hard time finding any detail present before the dragon obscures them not either exit frame right when the camera pans left slightly near the end or not re-appear with reasonably crisp detail after the dragon gets out of the way. Most of the mob of people are indistinct, but there is a woman in a lime green coat who is visible, and then obstructed by the dragon twice (beard and ribbon) and reappears fine. Unfortunately when dragon fully moves past she has been lost to frame right. There is another person in black holding a red satchel which is visible both before and after the dragon has passed. Nothing about the storefronts appear to change. The complex sign full of Chinese text (which might be gibberish text: it's highly stylized and I don't know Chinese) appears to survive the dragon passing without even any changes to the individual ideograms. There is also a red box shaped like a Chinese paper lantern with a single gold ideogram on it at the store entrance which spends most of the video obscured by the dragon and is still in the same location after it passes (though video artifacting makes it more challenging to verify that that ideogram is unchanged it certainly does not appear substantially different) What detail are you seeing that is different before and after the obstruction?