6 ms·

Joint 3D Face Reconstruction

124 points by tonylemesmer 8y ago

bsenftner 8y ago
Have at it, deep fake geeks...
tostitos1979 8y ago
Nice project! I've been dabbling in DL for about a year (skimmed Stanford CNN course, Silver's RNN course, Andrew Ng's old ML course, etc.). While I can recreate basic stuff like MNIST, I don't feel like I can attack a problem like Pose estimation yet. How long does it take? Is it about diving deep into a single problem? Is it worth doing a nanodegree to shore up gaps?
kpenc 8y ago
Wonder why it's written in Python 2.7 and not 3?
- mrtksn 8y ago
  In my experience with scientists, they just use the tools that know how to use and have available. So maybe they just got the set-up for 2.7 on their personal/work computers or maybe some tool/library is using 2.7?
mlboss 8y ago
University of Basel has some nice project for 3d Modeling. They also have a scala project for 3d face model construction using Gaussian Processes. All the recent 3d modeling papers use Basel Face Model. https://www.futurelearn.com/courses/statistical-shape-modelling/ https://www.futurelearn.com/courses/statistical-shape-modell... http://gravis.dmi.unibas.ch/PMM/ http://gravis.dmi.unibas.ch/PMM/ https://github.com/unibas-gravis/scalismo-faces https://github.com/unibas-gravis/scalismo-faces
codetrotter 8y ago
Good READMEs are important and this is a great example of a good README. The gif at the top alone instantly tells you a lot about what the project is about.
debt 8y ago
Only white people used as examples. Might be useful to literally diversify your dataset.
- lopmotr 8y ago
  They're also all humans. How diverse they go depends on the application people put it to. It's surely enough to prove the concept with one race and one species.
- wpietri 8y ago
  Absolutely. Even if one doesn't care about quality or inclusion, making this mistake can lead to big PR fails (and apparently lasting tech struggles): https://www.wired.com/story/when-it-comes-to-gorillas-google-photos-remains-blind/ https://www.wired.com/story/when-it-comes-to-gorillas-google...
- jahewson 8y ago
  This is somewhat ironic given that all five of the paper’s authors are East Asian.
  - TaylorAlexander 8y ago
    Fei Fei Li recently pointed out in a talk that if you search google images for “grandma” you get very biased results. They’re all white grandmothers! So it may be that the data sources the authors had are biased too. In any case this is an issue that should be addressed.
    - jurjenh 8y ago
      That made me wonder about what would happen if google started grouping concepts (eg words for grandmother in other languages will almost certainly give you differing results) rather than words... Possibly (probably) the cognitive load on people might lead to bad A/B test results, but it would be a curious thing to explore - cultural bias would get a bit of a beating I'd imagine.
    - kowdermeister 8y ago
pathartl 8y ago
New meme? https://github.com/YadiraF/PRNet/raw/master/Docs/images/reconstruct.jpg https://github.com/YadiraF/PRNet/raw/master/Docs/images/reco...
athenot 8y ago
I wonder if this can be extended for the whole body, so as to produce a 3D scan of the skin, and measure the evolution of moles and skin issues.
- polartx 8y ago
  >and measure the evolution of moles and skin issues be still my heart!
superbatfish 8y ago
The shortened HN title here is nonsensical. In this context, "joint" means "together" or "at the same time". The work's title, "Joint 3D Face Reconstruction and Dense Alignment", essentially means "Face Reconstruction with simultaneous Dense Alignment". The HN title should mention both parts, or remove the word "joint".
- tonylemesmer 8y ago
  yep - my mistake.
matte_black 8y ago
I just realized what this would be great for: Avatars in VR games. Snap a photo or video of your face and have it put straight into your character complete with animations. Excellent.
- Voltage 8y ago
  Check out an iPhone app called AffdexMe. It turns a photo into an animated 3d avatar in a few seconds. I work in AR/VR and we are evaluating whether the iphone can be used for decent facial capture.
  - tokyodude 8y ago
    Isn’t the answer already yes? https://m.youtube.com/watch?v=i51CizUXd7A https://m.youtube.com/watch?v=i51CizUXd7A I’d love to know what it’s short comings are vs other solutions
- canada_dry 8y ago
  But we all know what this'll be used for. Spoiler: pr0n
ddtaylor 8y ago
Looking through the repo, anyone help me find where they put the vertices data that maps to image set?

124 points by tonylemesmer 8y ago

Have at it, deep fake geeks...

tostitos1979 8y ago

Nice project! I've been dabbling in DL for about a year (skimmed Stanford CNN course, Silver's RNN course, Andrew Ng's old ML course, etc.). While I can recreate basic stuff like MNIST, I don't feel like I can attack a problem like Pose estimation yet. How long does it take? Is it about diving deep into a single problem? Is it worth doing a nanodegree to shore up gaps?

kpenc 8y ago

Wonder why it's written in Python 2.7 and not 3?

mrtksn 8y ago
In my experience with scientists, they just use the tools that know how to use and have available. So maybe they just got the set-up for 2.7 on their personal/work computers or maybe some tool/library is using 2.7?

mlboss 8y ago

University of Basel has some nice project for 3d Modeling. They also have a scala project for 3d face model construction using Gaussian Processes. All the recent 3d modeling papers use Basel Face Model. https://www.futurelearn.com/courses/statistical-shape-modelling/ https://www.futurelearn.com/courses/statistical-shape-modell... http://gravis.dmi.unibas.ch/PMM/ http://gravis.dmi.unibas.ch/PMM/ https://github.com/unibas-gravis/scalismo-faces https://github.com/unibas-gravis/scalismo-faces

codetrotter 8y ago

Good READMEs are important and this is a great example of a good README. The gif at the top alone instantly tells you a lot about what the project is about.

debt 8y ago

Only white people used as examples. Might be useful to literally diversify your dataset.

lopmotr 8y ago
They're also all humans. How diverse they go depends on the application people put it to. It's surely enough to prove the concept with one race and one species.
wpietri 8y ago
Absolutely. Even if one doesn't care about quality or inclusion, making this mistake can lead to big PR fails (and apparently lasting tech struggles): https://www.wired.com/story/when-it-comes-to-gorillas-google-photos-remains-blind/ https://www.wired.com/story/when-it-comes-to-gorillas-google...
jahewson 8y ago
This is somewhat ironic given that all five of the paper’s authors are East Asian.
- TaylorAlexander 8y ago
  Fei Fei Li recently pointed out in a talk that if you search google images for “grandma” you get very biased results. They’re all white grandmothers! So it may be that the data sources the authors had are biased too. In any case this is an issue that should be addressed.
  - jurjenh 8y ago
    That made me wonder about what would happen if google started grouping concepts (eg words for grandmother in other languages will almost certainly give you differing results) rather than words... Possibly (probably) the cognitive load on people might lead to bad A/B test results, but it would be a curious thing to explore - cultural bias would get a bit of a beating I'd imagine.
  - kowdermeister 8y ago

pathartl 8y ago

New meme? https://github.com/YadiraF/PRNet/raw/master/Docs/images/reconstruct.jpg https://github.com/YadiraF/PRNet/raw/master/Docs/images/reco...

athenot 8y ago

I wonder if this can be extended for the whole body, so as to produce a 3D scan of the skin, and measure the evolution of moles and skin issues.

polartx 8y ago
>and measure the evolution of moles and skin issues be still my heart!

superbatfish 8y ago

The shortened HN title here is nonsensical. In this context, "joint" means "together" or "at the same time". The work's title, "Joint 3D Face Reconstruction and Dense Alignment", essentially means "Face Reconstruction with simultaneous Dense Alignment". The HN title should mention both parts, or remove the word "joint".

tonylemesmer 8y ago
yep - my mistake.

matte_black 8y ago

I just realized what this would be great for: Avatars in VR games. Snap a photo or video of your face and have it put straight into your character complete with animations. Excellent.

Voltage 8y ago
Check out an iPhone app called AffdexMe. It turns a photo into an animated 3d avatar in a few seconds. I work in AR/VR and we are evaluating whether the iphone can be used for decent facial capture.
- tokyodude 8y ago
  Isn’t the answer already yes? https://m.youtube.com/watch?v=i51CizUXd7A https://m.youtube.com/watch?v=i51CizUXd7A I’d love to know what it’s short comings are vs other solutions
canada_dry 8y ago
But we all know what this'll be used for. Spoiler: pr0n

ddtaylor 8y ago

Looking through the repo, anyone help me find where they put the vertices data that maps to image set?