3 ms·
Why wouldn't they use 3d renderings as a large part of their training data set? You could have perfectly generated depth outputs generated alongside the image i
by ralusek 7y ago
Why wouldn't they use 3d renderings as a large part of their training data set? You could have perfectly generated depth outputs generated alongside the image input, and you could adjust things like focal length to all kinds of values that would make this able to understand how shifting items correlate to depth across a variety of focal lengths. To be honest I'm not even sure how they're training them with live footage, how are they even getting the depth maps from the training footage to begin with?
- pjc50 7y ago> we make use of an existing source of data for supervision: YouTube videos in which people imitate mannequins by freezing in a wide variety of natural poses, while a hand-held camera tours the scene. Because the entire scene is stationary (only the camera is moving), triangulation-based methods--like multi-view-stereo (MVS)--work, and we can get accurate depth maps for the entire scene including the people in it I suspect the reason for not using 3D rendering is the desire to cope with the noise and variability of real video.
- Fission 7y agoThe reason why is because there is no training data of the sort you describe out there. By using MVS-based approaches, they are able to get over the data hurdle by compiling a dataset of your average YouTube video, instead of creating 3D renderings that include dynamic people. Importantly, MVS is really quite accurate, and in many cases can be considered ground truth. Being able to forgo 3D renderings to use video only is almost certainly a reason why their results are so good.