4 ms·

That's right. It has to understand how objects move, and the boundaries between objects, what's part of a given object and what isn't, as well as how a given ob

That's right. It has to understand how objects move, and the boundaries between objects, what's part of a given object and what isn't, as well as how a given object will be affected by light when moving, shadows, etc. This is difficult to learn from data, even if you have a lot of it.

sangnoir 7y ago
> It has to understand how objects move[1], and the boundaries between objects[2], what's part of a given object and what isn't[3], as well as how a given object will be affected by light[4] when moving, shadows, etc. This is difficult to learn from data, even if you have a lot of it. Each one of the items you've listed has been severally accomplished by ML to varying levels of success (baseline being "workable", and more progress is being made.) I probably would have found better references had I spent longer than 5 minutes searching 1. https://www.youtube.com/watch?v=AGm3hF_BlYM https://www.youtube.com/watch?v=AGm3hF_BlYM 2. https://www.theverge.com/2019/12/9/20999646/google-arcore-augmented-reality-updates-occlusion-physics-depth https://www.theverge.com/2019/12/9/20999646/google-arcore-au... 3. Same as 2 :-). AR Occlusion requires both. Additionally many recent phones use ML models to fake depth-of-field; boundary between foreground objects has to be detected 4. Also see 2; Augmented Reality is hard. bonus: https://www.awn.com/animationworld/how-klaus-uniquely-combines-cg-lighting-techniques-traditional-2d-animation https://www.awn.com/animationworld/how-klaus-uniquely-combin...
- tachyonbeam 7y ago
  I work in deep learning research. In my opinion, pose estimation with one human centered in a frame is nowhere near the level of difficulty of constructing an accurate model of a complex a scene with multiple people and moving objects of various sizes. For instance, it wouldn't be so hard to detect that there is a train in this video, but to do upscaling correctly, you need a 3D model of this specific train. Yes, there exist deep learning models that try to do 3D reconstruction, and so far, I haven't seen anything that works robustly. Combining this information with an upscaling model is another challenge. It's not as easy as just plugging components together.

sangnoir 7y ago

> It has to understand how objects move[1], and the boundaries between objects[2], what's part of a given object and what isn't[3], as well as how a given object will be affected by light[4] when moving, shadows, etc. This is difficult to learn from data, even if you have a lot of it. Each one of the items you've listed has been severally accomplished by ML to varying levels of success (baseline being "workable", and more progress is being made.) I probably would have found better references had I spent longer than 5 minutes searching 1. https://www.youtube.com/watch?v=AGm3hF_BlYM https://www.youtube.com/watch?v=AGm3hF_BlYM 2. https://www.theverge.com/2019/12/9/20999646/google-arcore-augmented-reality-updates-occlusion-physics-depth https://www.theverge.com/2019/12/9/20999646/google-arcore-au... 3. Same as 2 :-). AR Occlusion requires both. Additionally many recent phones use ML models to fake depth-of-field; boundary between foreground objects has to be detected 4. Also see 2; Augmented Reality is hard. bonus: https://www.awn.com/animationworld/how-klaus-uniquely-combines-cg-lighting-techniques-traditional-2d-animation https://www.awn.com/animationworld/how-klaus-uniquely-combin...

tachyonbeam 7y ago
I work in deep learning research. In my opinion, pose estimation with one human centered in a frame is nowhere near the level of difficulty of constructing an accurate model of a complex a scene with multiple people and moving objects of various sizes. For instance, it wouldn't be so hard to detect that there is a train in this video, but to do upscaling correctly, you need a 3D model of this specific train. Yes, there exist deep learning models that try to do 3D reconstruction, and so far, I haven't seen anything that works robustly. Combining this information with an upscaling model is another challenge. It's not as easy as just plugging components together.