3 ms·

TLDR They use a neural net to process an image to find a similar known 3D model to the object in the image as well as parameters for how to deform that model to

TLDR They use a neural net to process an image to find a similar known 3D model to the object in the image as well as parameters for how to deform that model to be like the object in the image, and also then perform a linear combination of some related meshes (this is a follow up to their prior work "Compact Model Representation for 3D Reconstruction"). Pretty exciting work, but still far from being able to robustly do Image2Mesh. I happen to have worked on something quite similar, if any of you are curious: https://deformnet-site.github.io/DeformNet-website/ https://deformnet-site.github.io/DeformNet-website/

epberry 9y ago
Interesting, thanks for the link. I tried a few published methods for doing this a year or so and I found they were quite slow. As in, you'd rather just use a stereo camera to quickly generate a decent pointcloud than use a single view plus a network like these. What did you find your performance (ms/frame) was like?
- bsenftner 9y ago
  With separate dedicated identifiers and matching reconstruction pipelines, while only caring about a limited number of things, as in our specializing in faces, we can reconstruct 25 to 30 independent people, their heads and shoulder tops only, in real time on a 3.4 GHz i7. Note the reconstruction is accurate only as far as it matters for FR, as in we tend to ignore hair. So the models are bald, but they have the frame to frame facial expressions for everyone. Here's a gif of a degraded lighting test at 70 fps with one head doing a 3D reconstruction independently performed each frame: https://www.dropbox.com/s/0jtee4ofh2vdeg0/BlakeHeadMeshes.gif?dl=0 https://www.dropbox.com/s/0jtee4ofh2vdeg0/BlakeHeadMeshes.gi... I'm just visualizing the face, but the entire head and shoulder tops are reconstructed.
  - epberry 9y ago
    Ah this is cool too! I assume no neural networks are involved here? EDIT: why the changing colors in that gif?
    - bsenftner 9y ago
      The changing colors and grain are part of a degraded video while tracking the face test. The face detection and portions of the reconstruction are NNs.

epberry 9y ago

Interesting, thanks for the link. I tried a few published methods for doing this a year or so and I found they were quite slow. As in, you'd rather just use a stereo camera to quickly generate a decent pointcloud than use a single view plus a network like these. What did you find your performance (ms/frame) was like?

bsenftner 9y ago
With separate dedicated identifiers and matching reconstruction pipelines, while only caring about a limited number of things, as in our specializing in faces, we can reconstruct 25 to 30 independent people, their heads and shoulder tops only, in real time on a 3.4 GHz i7. Note the reconstruction is accurate only as far as it matters for FR, as in we tend to ignore hair. So the models are bald, but they have the frame to frame facial expressions for everyone. Here's a gif of a degraded lighting test at 70 fps with one head doing a 3D reconstruction independently performed each frame: https://www.dropbox.com/s/0jtee4ofh2vdeg0/BlakeHeadMeshes.gif?dl=0 https://www.dropbox.com/s/0jtee4ofh2vdeg0/BlakeHeadMeshes.gi... I'm just visualizing the face, but the entire head and shoulder tops are reconstructed.
- epberry 9y ago
  Ah this is cool too! I assume no neural networks are involved here? EDIT: why the changing colors in that gif?
  - bsenftner 9y ago
    The changing colors and grain are part of a degraded video while tracking the face test. The face detection and portions of the reconstruction are NNs.