3 ms·

Am I the only one that finds the "improved" version much worse? There are digital artifacts all over the place, like the steam at the beginning, the people's f

Am I the only one that finds the "improved" version much worse? There are digital artifacts all over the place, like the steam at the beginning, the people's faces, any place there is movement. I see that some things don't move as "jerky" but I think the original is far superior to watch. Edited to add: Here's a link to a quick comparison image. These aren't exactly the same frame or size, but close enough to see what I'm talking about. https://imgur.com/a/A3roH54 https://imgur.com/a/A3roH54

godson_drafty 7y ago
I agree, the improved version is worse. At first glance, it looks better than the original, but then numerous artifacts become apparent. For example, check out the little girl's face. It's jumbled in the "improved" version, but very clear in the original. It looks like most of the perceived improvement comes simply from the upgrade to 60fps, which is achieved trivially through interpolation. I wonder if the video would have gotten as much attention had AI not been mentioned.
tachyonbeam 7y ago
Someone posted a colorized[0] version and it had somewhat "jumpy" colors that change over time. I think one problem is you need temporal consistency for this to really work properly. If you pick a color for an object, or you upscale an object a certain way, it has to stay that way over a potentially unlimited number of frames. That's already hard to achieve because neural networks are typically bad at long-term dependency, but it's even more difficult if the object moves, because it implies that the neural network has to understand object identity and movement, which requires understanding physics, and what a train or a person is to begin with. At the moment, neural networks don't have sophisticated models of the world. [0]: https://www.youtube.com/watch?v=EqbOhqXHL7E https://www.youtube.com/watch?v=EqbOhqXHL7E
- sangnoir 7y ago
  >...because it implies that the neural network has to understand object identity and movement, which requires understanding physics, and what a train or a person is to begin with. I agree with most of what you said, but disagree with the need to know underlying fundamentals: most 5 year olds can throw and catch a ball without needing to understand calculus or Newton's laws of gravity.
  - mnowicki 7y ago
    they don't need to know the formulas or mathematical logic behind physics, but 5 year olds do have a working model of physics in their head. And AI wouldn't need to know the actual formulas either, but it would need some working model that owuld allow it to differentiate between those
    - tachyonbeam 7y ago
      That's right. It has to understand how objects move, and the boundaries between objects, what's part of a given object and what isn't, as well as how a given object will be affected by light when moving, shadows, etc. This is difficult to learn from data, even if you have a lot of it.
      - sangnoir 7y ago
        > It has to understand how objects move[1], and the boundaries between objects[2], what's part of a given object and what isn't[3], as well as how a given object will be affected by light[4] when moving, shadows, etc. This is difficult to learn from data, even if you have a lot of it. Each one of the items you've listed has been severally accomplished by ML to varying levels of success (baseline being "workable", and more progress is being made.) I probably would have found better references had I spent longer than 5 minutes searching 1. https://www.youtube.com/watch?v=AGm3hF_BlYM https://www.youtube.com/watch?v=AGm3hF_BlYM 2. https://www.theverge.com/2019/12/9/20999646/google-arcore-augmented-reality-updates-occlusion-physics-depth https://www.theverge.com/2019/12/9/20999646/google-arcore-au... 3. Same as 2 :-). AR Occlusion requires both. Additionally many recent phones use ML models to fake depth-of-field; boundary between foreground objects has to be detected 4. Also see 2; Augmented Reality is hard. bonus: https://www.awn.com/animationworld/how-klaus-uniquely-combines-cg-lighting-techniques-traditional-2d-animation https://www.awn.com/animationworld/how-klaus-uniquely-combin...
        tachyonbeam 7y ago
        I work in deep learning research. In my opinion, pose estimation with one human centered in a frame is nowhere near the level of difficulty of constructing an accurate model of a complex a scene with multiple people and moving objects of various sizes. For instance, it wouldn't be so hard to detect that there is a train in this video, but to do upscaling correctly, you need a 3D model of this specific train. Yes, there exist deep learning models that try to do 3D reconstruction, and so far, I haven't seen anything that works robustly. Combining this information with an upscaling model is another challenge. It's not as easy as just plugging components together.
te_platt 7y ago
In most ways, yes. For me the first 5 seconds or so were fantastic. It felt like seeing a behind the scenes "How it was made" documentary where I was seeing the actors out of character. Or like it removed the sense of being from another era. Soon the artifacts and warping became distracting, and then watching the original put it back in time. Maybe it was a sense of nostalgia but overall the original seemed better to me.

godson_drafty 7y ago

I agree, the improved version is worse. At first glance, it looks better than the original, but then numerous artifacts become apparent. For example, check out the little girl's face. It's jumbled in the "improved" version, but very clear in the original. It looks like most of the perceived improvement comes simply from the upgrade to 60fps, which is achieved trivially through interpolation. I wonder if the video would have gotten as much attention had AI not been mentioned.

tachyonbeam 7y ago

Someone posted a colorized[0] version and it had somewhat "jumpy" colors that change over time. I think one problem is you need temporal consistency for this to really work properly. If you pick a color for an object, or you upscale an object a certain way, it has to stay that way over a potentially unlimited number of frames. That's already hard to achieve because neural networks are typically bad at long-term dependency, but it's even more difficult if the object moves, because it implies that the neural network has to understand object identity and movement, which requires understanding physics, and what a train or a person is to begin with. At the moment, neural networks don't have sophisticated models of the world. [0]: https://www.youtube.com/watch?v=EqbOhqXHL7E https://www.youtube.com/watch?v=EqbOhqXHL7E

sangnoir 7y ago
>...because it implies that the neural network has to understand object identity and movement, which requires understanding physics, and what a train or a person is to begin with. I agree with most of what you said, but disagree with the need to know underlying fundamentals: most 5 year olds can throw and catch a ball without needing to understand calculus or Newton's laws of gravity.
- mnowicki 7y ago
  they don't need to know the formulas or mathematical logic behind physics, but 5 year olds do have a working model of physics in their head. And AI wouldn't need to know the actual formulas either, but it would need some working model that owuld allow it to differentiate between those
  - tachyonbeam 7y ago
    That's right. It has to understand how objects move, and the boundaries between objects, what's part of a given object and what isn't, as well as how a given object will be affected by light when moving, shadows, etc. This is difficult to learn from data, even if you have a lot of it.
    - sangnoir 7y ago
      > It has to understand how objects move[1], and the boundaries between objects[2], what's part of a given object and what isn't[3], as well as how a given object will be affected by light[4] when moving, shadows, etc. This is difficult to learn from data, even if you have a lot of it. Each one of the items you've listed has been severally accomplished by ML to varying levels of success (baseline being "workable", and more progress is being made.) I probably would have found better references had I spent longer than 5 minutes searching 1. https://www.youtube.com/watch?v=AGm3hF_BlYM https://www.youtube.com/watch?v=AGm3hF_BlYM 2. https://www.theverge.com/2019/12/9/20999646/google-arcore-augmented-reality-updates-occlusion-physics-depth https://www.theverge.com/2019/12/9/20999646/google-arcore-au... 3. Same as 2 :-). AR Occlusion requires both. Additionally many recent phones use ML models to fake depth-of-field; boundary between foreground objects has to be detected 4. Also see 2; Augmented Reality is hard. bonus: https://www.awn.com/animationworld/how-klaus-uniquely-combines-cg-lighting-techniques-traditional-2d-animation https://www.awn.com/animationworld/how-klaus-uniquely-combin...
      - tachyonbeam 7y ago
        I work in deep learning research. In my opinion, pose estimation with one human centered in a frame is nowhere near the level of difficulty of constructing an accurate model of a complex a scene with multiple people and moving objects of various sizes. For instance, it wouldn't be so hard to detect that there is a train in this video, but to do upscaling correctly, you need a 3D model of this specific train. Yes, there exist deep learning models that try to do 3D reconstruction, and so far, I haven't seen anything that works robustly. Combining this information with an upscaling model is another challenge. It's not as easy as just plugging components together.

te_platt 7y ago

In most ways, yes. For me the first 5 seconds or so were fantastic. It felt like seeing a behind the scenes "How it was made" documentary where I was seeing the actors out of character. Or like it removed the sense of being from another era. Soon the artifacts and warping became distracting, and then watching the original put it back in time. Maybe it was a sense of nostalgia but overall the original seemed better to me.