11 ms·

Total Moving Face Reconstruction

308 points by mxfh 12y ago

Aqwis 12y ago
Very impressive. How large does the photo collection of the individual have to be to achieve results like those in the video?
- anigbrowl 12y ago
  The source videos they used as examples are about 600 frames each, which is ~40 seconds of video. For a politician or professional actor this would be easy to achieve. Likewise if you wanted a good 3d model of yourself or a friend you could do it cheaply using this method.
  - Aqwis 12y ago
    I may be misreading the paper, but as far as I understand you need a collection of photos in addition to the video.
    - eric_h 12y ago
      and judging by their selection of subjects, I would presume the accompanying photo library would need to be quite large.
    - anigbrowl 12y ago
      I thought that too, but then they have some ambiguous sentences like 'In this paper we target high detail reconstruction from a single video captured in the wild, i.e., under uncontrolled imaging conditions,' where it seems as if they treat that as their input library. Other parts of the paper support your interpretation. It's annoyingly vague. ...but still, outstanding work even if it requires manual calibration like curation of the input set.
hunvreus 12y ago
Can't help but think of "The Running Man" watching Schwarzenegger's face being rendered in 3D. It's terrifying to think that in the next 5 to 10 years we won't be able to distinguish a forged, high definition video of pretty much anybody.
- canadaj 12y ago
  Terrifying would be continuing to take everything we see as fact in 5 to 10 years. I think we are in a special time where if you see a video of pretty much anybody, you can usually tell right away if it's fake or not. Sometimes it's hard, but it can usually be debunked. Before that, you really couldn't fake any videos without it being obvious. In 5 to 10 years, hopefully we will have learned to never ever ever take anything we see as fact, because we absolutely will not be able to distinguish rendered video from the real thing.
  - anigbrowl 12y ago
    True, but the downside of this is that anyone caught in the act will use this as a defense of first resort - it'll be the 2020's equivalent of 'my twitter account was hacked' until we establish some sort of reliable ELA tool to grade source material. Indeed, as there are more and more cameras around (including autonomous ones of increasingly tiny size) imagery of the videographer will probably become a major authentication factor.
    - canadaj 12y ago
      I believe everything you have said is true, especially with cameras becoming smaller and more common. However, I'll still be optimistic and hope that with the increasing number of cameras, people will be less likely to engage in activities where they shouldn't. The opposite argument would be that with the increasing technology to fake such an activity, the amount of 'my twitter account was hacked' incidents will rise. I guess only time will tell?
      - guelo 12y ago
daniel_reetz 12y ago
I'd like to ask the authors how they managed to do such great/natural looking reconstructions of the eyes. Eyes are tough because they're naturally specular, transparent in places, and refractive.
- danbruc 12y ago
  After quickly reading the paper, I think they did nothing special for the eyes; it's probably just the result of the final refinement step. The result looses a bit of its magic, though, when you read in more detail how it works. But it remains impressive.
- AJ007 12y ago
  It is rendering the eyes as geometry, take a look at the limitations at end of the video, that gives a good demonstration of what is going on.
imaginenore 12y ago
This kind of advancements is one of the reasons I don't post photos of myself online. In a few years we will be capable of making videos with anybody's face replaced with anybody else's. It will be trivial to produce a fake video that can cause all kinds of legal troubles. And yes, I realize it's even possible now, but with all the new algos and software coming out it will be easy enough for somebody to just mess with people's lives for fun.
- maxerickson 12y ago
  If it's trivial, it will probably undermine the utility of video in legal circumstances more than it impacts the people depicted in the videos.
  - r00fus 12y ago
    You assume legal framework will keep up to pace with technology with no lag. I trust that, due to jurisprudence model, the legal framework will always have some lag[1] after game-changing technology becomes mainstream... sometimes this can be significant. [1] disclaimer: active legislation efforts can move faster than tech - especially due to lobbying.
    - anigbrowl 12y ago
      In theory it doesn't have to - there are already legal procedures for qualifying and admitting expert testimony at trial that work quite well (see http://en.wikipedia.org/wiki/Daubert_standard http://en.wikipedia.org/wiki/Daubert_standard). So if you were in jeopardy primarily due to video evidence which you knew to be faked then you could bring a machine vision or CG expert into court to demonstrate the unreliability of the video evidence. But this is not to say that our system for handling forensic evidence works great - it doesn't sadly, and part of the problem is that juries are often reduced to trying to weigh the credibility of competing expert testators without rigorous standards of measurement or terminology. You'll probably be interested in this: http://www8.nationalacademies.org/onpinews/newsitem.aspx?RecordID=12589 http://www8.nationalacademies.org/onpinews/newsitem.aspx?Rec...
- 12y ago
anigbrowl 12y ago
Somewhat off-topic, but I wonder why facial recognition/modeling experts seem to persistently ignore ears and jawline. As someone who works in film and does some picture editing (though it's not my primary skillset), ears are just as individual as other parts of the face, and they're one of the trickiest things for makeup artists to work on. As CG in movies and videogames keeps improving, my suspension of disbelief is often broken by noticing problems with the ears, eg watching a CG anime film and noticing that everyone has the same ear shape.
- sjtrny 12y ago
  In this case it is likely because ears can be obscured by hair. If you notice this technique has some difficulty just with shadows so they limit it to the face.
- swartkrans 12y ago
  > As CG in movies and videogames keeps improving, my suspension of disbelief is often broken I have never really seen CG portrayal of people in live action films that didn't break suspension of disbelief, it always falls right into uncanny valley stiffness. The exception being Avatar, which for whatever reason doesn't seem to have had the problems other films have.
  - Swizec 12y ago
    > which for whatever reason doesn't seem to have had the problems other films have. By not CGI-ing humans. That's enough for you to stop judging what you're seeing by the same criteria as you do real life. They wisely chose to stay well to the left-side of uncanny valley instead of trying to jump over to the right-side. Look it up. It's also why Wall-E feels so human.
  - djloche 12y ago
    I would wager that this is because most of Avatar is not live-action, but CG, with real people added in. Spend enough money/time and live-action people fit in really well to CG environments.
  - 12y ago
aresant 12y ago
Technology likes this makes me wonder how long of a shelf-life video "evidence" has. Or perhaps these same algos will also provide utility in detecting / decoding "fakes", sort of like edge-tracking / error level analysis etc today.
SnowProblem 12y ago
This will be huge for VR.
sabalaba 12y ago
3D reconstruction is used in state-of-the-art facial recognition as well.[1] Essentially you reconstruct the face in 3D, rotate the 3D model to the front, project it back into 2 dimensions, and then feed it through a CNN with deep architecture. Because this gives you very good alignment, you can do tricks like not having shared weights across the entire image. That is, each section of the input vector is known to correspond to a certain part of the face and thus can learn unique parameters that are well suited for that specific region. The paper claims that it takes about 105 seconds to render a single frame. So one second of 30 fps video would take about 52 minutes to render. I would have to read more in depth to see what kind of savings can be had by sharing information across frames. (The paper also doesn't mention the use of GPU acceleration.) [1] https://www.facebook.com/publications/546316888800776/ https://www.facebook.com/publications/546316888800776/
- danbruc 12y ago
  Roughly they use a model of the face, render it, compare the source image and the rendered image, estimate changes of position, orientation and deformation for the model that will yield a better match and then just repeat this until the result is good enough. While you probably can exploit temporal coherence between frames the process is inherently pretty expensive due to its iterative nature. But it may also be relatively easy to parallelize bbecause of this.
- deleted 12y ago
  [deleted]
menriquez 12y ago
say goodbye to real actors soon...amazing.
debt 12y ago
I've been increasingly interested in the Face. Human beings must have some incredible mental calculations going on when parsing a face. We're an evolved species that use the face as a form of communication. I love the attached video in the link because it isolates perfectly the face. If you look closely you can see these tiny minute combinations within the face as each person talks; the eyes shifting, the face rotating, looking in various directions, the forehead crunching, the eyebrows raising, smiling, etc. All of these "cues" combine to create a message that we interpret instantly. The face has inspired me lately to read more into this subject as it seems, at least on the surface, to be an extremely complex innate human ability; facial recognition.
- Throwaway12830 12y ago
  It really is incredible. I mean, in the original link, look at the image of Arnold used for the video before playing. It's a blurry, greyscale section of his face. Nonetheless, most people could easily recognize that face as being him. Billions of people in this world, we all have a very similar facial structure with two eyes, a nose and a mouth, and yet you can recognize that small blurry face in a fraction of a second. I imagine it's similar with animals. You can have thousands of birds in a flock, and they can recognize their mate instantly. To us, we'd have to carefully analyze the birds for days or weeks to make that same match.
  - 31reasons 12y ago
    What amazing is that scientists discovered that brain allocates one single neuron fires for each recognized face. For example, You have a one special neuron in your brain that fires when you see Bill Clinton's face. And that exact neuron fires no matter what picture of Bill Clinton you happened to see.
    - lutusp 12y ago
      > What amazing is that scientists discovered that brain allocates one single neuron fires for each recognized face. This is nonsense, citation needed. > For example, You have a one special neuron in your brain that fires when you see Bill Clinton's face. This is New Age science, i.e. not science at all. It's a myth. One neuron is equivalent to one bit in a computer. One bit of information is insufficient to distinguish between faces. > And that exact neuron fires no matter what picture of Bill Clinton you happened to see. Maybe you could learn a tiny bit of neuroscience before spreading this kind of nonsense.
tantalor 12y ago
Tragic they removed the verbal audio from the demo video. It would have been much easier to judge the visual accuracy if the reconstructed lip motion were combined the original sound.
31reasons 12y ago
This + Virtual Gesture Tracking + VR = Virtual Meetings
Harshit15 12y ago
This can help a lot in recreating the faces of avatars, in animated movies and games. They have tough time tracking facial details using small markers. I was wondering using shadow and shine removal to solve the issues shown in end. An example here is implemented by these autonomous car designers detailed as shadow correction: http://www.igvc.org/design/2013/US%20Naval%20Academy.pdf http://www.igvc.org/design/2013/US%20Naval%20Academy.pdf
polskibus 12y ago
The question that burns me is - when will we see this amazing algorithm implemented as part of OpenCV ?
macca321 12y ago
I think I'm missing the point. Why are all the reconstructed videos from the same angle? It would demonstrate it better if they repositioned the camera.
- omellet 12y ago
  If you watch the George Bush segment, it shows the reconstructed video from a different angle (straight-on, eliminating the head movement).
- blt 12y ago
  > We note that our method does not guarantee an accurate fit to ground-truth geometry, as the shape of the face may change in each frame and single-image cues are not sufficient for this purpose. Rather, we seek to produce a reasonably convincing model (leveraging all available imagery) which optimally fits the image information in each frame. This makes it seem like the mesh produced for each frame is specially deformed to render optimally for that camera angle, and it wouldn't necessarily look perfect from other angles.
igriffer 12y ago
Hi! Anybody have some sources? I want to touch this method =) This 3D reconstructions are the best material for the face recognition!
Htsthbjig 12y ago
Quite dangerous what this technology will mean in the future, they could manufacture evidence against you, publish it, and then let the masses lynch you. I think this is what happened to the recent decapitation videos, they were reconstructed from home videos. IMO the videos with the people dead in the floor are true, but the videos where they talk are staged. Today we know there was a CIA team whose job was faking videos of Osama Bin laden: http://blog.washingtonpost.com/spy-talk/2010/05/cia_group_had_wacky_ideas_to_d.html http://blog.washingtonpost.com/spy-talk/2010/05/cia_group_ha... Remember Osama Bin Ladem appeared and disappeared according to US army interest at the time, finally ending in very strange circumstances(and being buried on the ocean, not letting anyone else interantionally to confirm(by DNA) he was Osama). For me it is staged because current technology could synthesize a voice only if there are not strong emotions. The same happens with the voice. With strong emotions it becomes very easy for familiars and friends to notice as people do specific gestures and most of them are not recorded in video. That people are perfectly calm before dying I could understand, but that they do while saying exactly what their captors want I can't. Also, before the videos most of the population in UK did not want to go to war, after the videos(with a UK native), most of them support war, quo prodis?
phkahler 12y ago
I like that they show cases where it has problems. It's very much "here's what we can do, and here's where it doesn't work." There is no hype, no claims of "novelty", no speculation on uses, just results. I wish this were far more common.
- blt 12y ago
  I agree, but I think claims of "novelty" in academic papers just mean "we don't think anyone has done it this way before." It's almost boilerplate.
- Iv 12y ago
  I still would like to see the reconstruction from another angle than the original. Depending on how much you accept to cheat, it is not too hard to get a good result from that anfle only.
  - pimlottc 12y ago
    They demonstrate the reconstruction at different angles in the video @1:49
bsenftner 12y ago
I run a startup specializing in this space called the 3D Avatar Store (www.3d-avatar-store.com). 3D Reconstruction of human faces is literally on the edge of mainstream. I'm betting on it, personally. Our system is similar as theirs, but more general: we laser scanned 300,000 real people and then associated each laser scan with dozens of photos of that person taken from different angles, lighting conditions and expressions. That data set was then used for a neural net training - actually a pipeline of neural nets. We can accept 1 photo and get back a good quality 3D model, or a series of photos and get better quality, or HD quality video and get back frame by frame, in expression reconstructions just like their solution. In fact, our system is able to recover 36 people per video feed in real time, as well as handle 4 video feeds at once. We don't need as much reference information as they do, because we trained our system to generally understand the human facial form, rather than their solution that operates in isolation for a single reconstruction operation. Our current system is targeted as a WebAPI for games and serious simulations - enabling 3rd parties to implement "put yourself in the game" functionality. As such we have 3 different geometry outputs aimed at game/simulation developers. We also do facial recognition, and we have a special "forensic" output for that. Our current "best output" is purposely "Pixar like" rather than realistic. Taking them realistic tends to freak people out - especially women (seems like our culture has trained women to have an idealized self image, and when presented with their non-mirror true form, they don't like it.) You can learn more at these links: https://3d-avatar-store.com/Web-API-Features-May-2014 https://3d-avatar-store.com/Web-API-Features-May-2014 https://3d-avatar-store.com/3D-Avatar-Creation-walkthru https://3d-avatar-store.com/3D-Avatar-Creation-walkthru https://3d-avatar-store.com/New-Face-Finder https://3d-avatar-store.com/New-Face-Finder
- bsenftner 12y ago
  And it looks like our site is under attack at the moment. If it does not load, give it a beat. We are seeing thousands of hits per second from eastern Europe right now. (Thank you Federal Reserve quality hardware firewall!)
- kemelmi 12y ago
  Very nice! Our work is different though: we create high detail 3d moving 3d models from youtube videos -- without any manual interaction (looks like there is quite a bit of interaction in creation of an avatar on 3d-avatar-store) Ira http://homes.cs.washington.edu/~kemelmi/ http://homes.cs.washington.edu/~kemelmi/

308 points by mxfh 12y ago

Aqwis 12y ago

Very impressive. How large does the photo collection of the individual have to be to achieve results like those in the video?

anigbrowl 12y ago
The source videos they used as examples are about 600 frames each, which is ~40 seconds of video. For a politician or professional actor this would be easy to achieve. Likewise if you wanted a good 3d model of yourself or a friend you could do it cheaply using this method.
- Aqwis 12y ago
  I may be misreading the paper, but as far as I understand you need a collection of photos in addition to the video.
  - eric_h 12y ago
    and judging by their selection of subjects, I would presume the accompanying photo library would need to be quite large.
  - anigbrowl 12y ago
    I thought that too, but then they have some ambiguous sentences like 'In this paper we target high detail reconstruction from a single video captured in the wild, i.e., under uncontrolled imaging conditions,' where it seems as if they treat that as their input library. Other parts of the paper support your interpretation. It's annoyingly vague. ...but still, outstanding work even if it requires manual calibration like curation of the input set.

hunvreus 12y ago

Can't help but think of "The Running Man" watching Schwarzenegger's face being rendered in 3D. It's terrifying to think that in the next 5 to 10 years we won't be able to distinguish a forged, high definition video of pretty much anybody.

canadaj 12y ago
Terrifying would be continuing to take everything we see as fact in 5 to 10 years. I think we are in a special time where if you see a video of pretty much anybody, you can usually tell right away if it's fake or not. Sometimes it's hard, but it can usually be debunked. Before that, you really couldn't fake any videos without it being obvious. In 5 to 10 years, hopefully we will have learned to never ever ever take anything we see as fact, because we absolutely will not be able to distinguish rendered video from the real thing.
- anigbrowl 12y ago
  True, but the downside of this is that anyone caught in the act will use this as a defense of first resort - it'll be the 2020's equivalent of 'my twitter account was hacked' until we establish some sort of reliable ELA tool to grade source material. Indeed, as there are more and more cameras around (including autonomous ones of increasingly tiny size) imagery of the videographer will probably become a major authentication factor.
  - canadaj 12y ago
    I believe everything you have said is true, especially with cameras becoming smaller and more common. However, I'll still be optimistic and hope that with the increasing number of cameras, people will be less likely to engage in activities where they shouldn't. The opposite argument would be that with the increasing technology to fake such an activity, the amount of 'my twitter account was hacked' incidents will rise. I guess only time will tell?
    - guelo 12y ago

daniel_reetz 12y ago

I'd like to ask the authors how they managed to do such great/natural looking reconstructions of the eyes. Eyes are tough because they're naturally specular, transparent in places, and refractive.

danbruc 12y ago
After quickly reading the paper, I think they did nothing special for the eyes; it's probably just the result of the final refinement step. The result looses a bit of its magic, though, when you read in more detail how it works. But it remains impressive.
AJ007 12y ago
It is rendering the eyes as geometry, take a look at the limitations at end of the video, that gives a good demonstration of what is going on.

imaginenore 12y ago

This kind of advancements is one of the reasons I don't post photos of myself online. In a few years we will be capable of making videos with anybody's face replaced with anybody else's. It will be trivial to produce a fake video that can cause all kinds of legal troubles. And yes, I realize it's even possible now, but with all the new algos and software coming out it will be easy enough for somebody to just mess with people's lives for fun.

maxerickson 12y ago
If it's trivial, it will probably undermine the utility of video in legal circumstances more than it impacts the people depicted in the videos.
- r00fus 12y ago
  You assume legal framework will keep up to pace with technology with no lag. I trust that, due to jurisprudence model, the legal framework will always have some lag[1] after game-changing technology becomes mainstream... sometimes this can be significant. [1] disclaimer: active legislation efforts can move faster than tech - especially due to lobbying.
  - anigbrowl 12y ago
    In theory it doesn't have to - there are already legal procedures for qualifying and admitting expert testimony at trial that work quite well (see http://en.wikipedia.org/wiki/Daubert_standard http://en.wikipedia.org/wiki/Daubert_standard). So if you were in jeopardy primarily due to video evidence which you knew to be faked then you could bring a machine vision or CG expert into court to demonstrate the unreliability of the video evidence. But this is not to say that our system for handling forensic evidence works great - it doesn't sadly, and part of the problem is that juries are often reduced to trying to weigh the credibility of competing expert testators without rigorous standards of measurement or terminology. You'll probably be interested in this: http://www8.nationalacademies.org/onpinews/newsitem.aspx?RecordID=12589 http://www8.nationalacademies.org/onpinews/newsitem.aspx?Rec...
12y ago

anigbrowl 12y ago

Somewhat off-topic, but I wonder why facial recognition/modeling experts seem to persistently ignore ears and jawline. As someone who works in film and does some picture editing (though it's not my primary skillset), ears are just as individual as other parts of the face, and they're one of the trickiest things for makeup artists to work on. As CG in movies and videogames keeps improving, my suspension of disbelief is often broken by noticing problems with the ears, eg watching a CG anime film and noticing that everyone has the same ear shape.

sjtrny 12y ago
In this case it is likely because ears can be obscured by hair. If you notice this technique has some difficulty just with shadows so they limit it to the face.
swartkrans 12y ago
> As CG in movies and videogames keeps improving, my suspension of disbelief is often broken I have never really seen CG portrayal of people in live action films that didn't break suspension of disbelief, it always falls right into uncanny valley stiffness. The exception being Avatar, which for whatever reason doesn't seem to have had the problems other films have.
- Swizec 12y ago
  > which for whatever reason doesn't seem to have had the problems other films have. By not CGI-ing humans. That's enough for you to stop judging what you're seeing by the same criteria as you do real life. They wisely chose to stay well to the left-side of uncanny valley instead of trying to jump over to the right-side. Look it up. It's also why Wall-E feels so human.
- djloche 12y ago
  I would wager that this is because most of Avatar is not live-action, but CG, with real people added in. Spend enough money/time and live-action people fit in really well to CG environments.
- 12y ago

aresant 12y ago

Technology likes this makes me wonder how long of a shelf-life video "evidence" has. Or perhaps these same algos will also provide utility in detecting / decoding "fakes", sort of like edge-tracking / error level analysis etc today.

SnowProblem 12y ago

This will be huge for VR.

sabalaba 12y ago

3D reconstruction is used in state-of-the-art facial recognition as well.[1] Essentially you reconstruct the face in 3D, rotate the 3D model to the front, project it back into 2 dimensions, and then feed it through a CNN with deep architecture. Because this gives you very good alignment, you can do tricks like not having shared weights across the entire image. That is, each section of the input vector is known to correspond to a certain part of the face and thus can learn unique parameters that are well suited for that specific region. The paper claims that it takes about 105 seconds to render a single frame. So one second of 30 fps video would take about 52 minutes to render. I would have to read more in depth to see what kind of savings can be had by sharing information across frames. (The paper also doesn't mention the use of GPU acceleration.) [1] https://www.facebook.com/publications/546316888800776/ https://www.facebook.com/publications/546316888800776/

danbruc 12y ago
Roughly they use a model of the face, render it, compare the source image and the rendered image, estimate changes of position, orientation and deformation for the model that will yield a better match and then just repeat this until the result is good enough. While you probably can exploit temporal coherence between frames the process is inherently pretty expensive due to its iterative nature. But it may also be relatively easy to parallelize bbecause of this.
deleted 12y ago
[deleted]

menriquez 12y ago

say goodbye to real actors soon...amazing.

debt 12y ago

I've been increasingly interested in the Face. Human beings must have some incredible mental calculations going on when parsing a face. We're an evolved species that use the face as a form of communication. I love the attached video in the link because it isolates perfectly the face. If you look closely you can see these tiny minute combinations within the face as each person talks; the eyes shifting, the face rotating, looking in various directions, the forehead crunching, the eyebrows raising, smiling, etc. All of these "cues" combine to create a message that we interpret instantly. The face has inspired me lately to read more into this subject as it seems, at least on the surface, to be an extremely complex innate human ability; facial recognition.

Throwaway12830 12y ago
It really is incredible. I mean, in the original link, look at the image of Arnold used for the video before playing. It's a blurry, greyscale section of his face. Nonetheless, most people could easily recognize that face as being him. Billions of people in this world, we all have a very similar facial structure with two eyes, a nose and a mouth, and yet you can recognize that small blurry face in a fraction of a second. I imagine it's similar with animals. You can have thousands of birds in a flock, and they can recognize their mate instantly. To us, we'd have to carefully analyze the birds for days or weeks to make that same match.
- 31reasons 12y ago
  What amazing is that scientists discovered that brain allocates one single neuron fires for each recognized face. For example, You have a one special neuron in your brain that fires when you see Bill Clinton's face. And that exact neuron fires no matter what picture of Bill Clinton you happened to see.
  - lutusp 12y ago
    > What amazing is that scientists discovered that brain allocates one single neuron fires for each recognized face. This is nonsense, citation needed. > For example, You have a one special neuron in your brain that fires when you see Bill Clinton's face. This is New Age science, i.e. not science at all. It's a myth. One neuron is equivalent to one bit in a computer. One bit of information is insufficient to distinguish between faces. > And that exact neuron fires no matter what picture of Bill Clinton you happened to see. Maybe you could learn a tiny bit of neuroscience before spreading this kind of nonsense.

tantalor 12y ago

Tragic they removed the verbal audio from the demo video. It would have been much easier to judge the visual accuracy if the reconstructed lip motion were combined the original sound.

31reasons 12y ago

This + Virtual Gesture Tracking + VR = Virtual Meetings

Harshit15 12y ago

This can help a lot in recreating the faces of avatars, in animated movies and games. They have tough time tracking facial details using small markers. I was wondering using shadow and shine removal to solve the issues shown in end. An example here is implemented by these autonomous car designers detailed as shadow correction: http://www.igvc.org/design/2013/US%20Naval%20Academy.pdf http://www.igvc.org/design/2013/US%20Naval%20Academy.pdf

polskibus 12y ago

The question that burns me is - when will we see this amazing algorithm implemented as part of OpenCV ?

macca321 12y ago

I think I'm missing the point. Why are all the reconstructed videos from the same angle? It would demonstrate it better if they repositioned the camera.

omellet 12y ago
If you watch the George Bush segment, it shows the reconstructed video from a different angle (straight-on, eliminating the head movement).
blt 12y ago
> We note that our method does not guarantee an accurate fit to ground-truth geometry, as the shape of the face may change in each frame and single-image cues are not sufficient for this purpose. Rather, we seek to produce a reasonably convincing model (leveraging all available imagery) which optimally fits the image information in each frame. This makes it seem like the mesh produced for each frame is specially deformed to render optimally for that camera angle, and it wouldn't necessarily look perfect from other angles.

igriffer 12y ago

Hi! Anybody have some sources? I want to touch this method =) This 3D reconstructions are the best material for the face recognition!

Htsthbjig 12y ago

Quite dangerous what this technology will mean in the future, they could manufacture evidence against you, publish it, and then let the masses lynch you. I think this is what happened to the recent decapitation videos, they were reconstructed from home videos. IMO the videos with the people dead in the floor are true, but the videos where they talk are staged. Today we know there was a CIA team whose job was faking videos of Osama Bin laden: http://blog.washingtonpost.com/spy-talk/2010/05/cia_group_had_wacky_ideas_to_d.html http://blog.washingtonpost.com/spy-talk/2010/05/cia_group_ha... Remember Osama Bin Ladem appeared and disappeared according to US army interest at the time, finally ending in very strange circumstances(and being buried on the ocean, not letting anyone else interantionally to confirm(by DNA) he was Osama). For me it is staged because current technology could synthesize a voice only if there are not strong emotions. The same happens with the voice. With strong emotions it becomes very easy for familiars and friends to notice as people do specific gestures and most of them are not recorded in video. That people are perfectly calm before dying I could understand, but that they do while saying exactly what their captors want I can't. Also, before the videos most of the population in UK did not want to go to war, after the videos(with a UK native), most of them support war, quo prodis?

phkahler 12y ago

I like that they show cases where it has problems. It's very much "here's what we can do, and here's where it doesn't work." There is no hype, no claims of "novelty", no speculation on uses, just results. I wish this were far more common.

blt 12y ago
I agree, but I think claims of "novelty" in academic papers just mean "we don't think anyone has done it this way before." It's almost boilerplate.
Iv 12y ago
I still would like to see the reconstruction from another angle than the original. Depending on how much you accept to cheat, it is not too hard to get a good result from that anfle only.
- pimlottc 12y ago
  They demonstrate the reconstruction at different angles in the video @1:49

bsenftner 12y ago

I run a startup specializing in this space called the 3D Avatar Store (www.3d-avatar-store.com). 3D Reconstruction of human faces is literally on the edge of mainstream. I'm betting on it, personally. Our system is similar as theirs, but more general: we laser scanned 300,000 real people and then associated each laser scan with dozens of photos of that person taken from different angles, lighting conditions and expressions. That data set was then used for a neural net training - actually a pipeline of neural nets. We can accept 1 photo and get back a good quality 3D model, or a series of photos and get better quality, or HD quality video and get back frame by frame, in expression reconstructions just like their solution. In fact, our system is able to recover 36 people per video feed in real time, as well as handle 4 video feeds at once. We don't need as much reference information as they do, because we trained our system to generally understand the human facial form, rather than their solution that operates in isolation for a single reconstruction operation. Our current system is targeted as a WebAPI for games and serious simulations - enabling 3rd parties to implement "put yourself in the game" functionality. As such we have 3 different geometry outputs aimed at game/simulation developers. We also do facial recognition, and we have a special "forensic" output for that. Our current "best output" is purposely "Pixar like" rather than realistic. Taking them realistic tends to freak people out - especially women (seems like our culture has trained women to have an idealized self image, and when presented with their non-mirror true form, they don't like it.) You can learn more at these links: https://3d-avatar-store.com/Web-API-Features-May-2014 https://3d-avatar-store.com/Web-API-Features-May-2014 https://3d-avatar-store.com/3D-Avatar-Creation-walkthru https://3d-avatar-store.com/3D-Avatar-Creation-walkthru https://3d-avatar-store.com/New-Face-Finder https://3d-avatar-store.com/New-Face-Finder

bsenftner 12y ago
And it looks like our site is under attack at the moment. If it does not load, give it a beat. We are seeing thousands of hits per second from eastern Europe right now. (Thank you Federal Reserve quality hardware firewall!)
kemelmi 12y ago
Very nice! Our work is different though: we create high detail 3d moving 3d models from youtube videos -- without any manual interaction (looks like there is quite a bit of interaction in creation of an avatar on 3d-avatar-store) Ira http://homes.cs.washington.edu/~kemelmi/ http://homes.cs.washington.edu/~kemelmi/