9 ms·

Very cool, but very creepy, open source project

456 points by monochromatic 15y ago

chaostheory 15y ago
Here's where his broken link meant to go: http://methodart.blogspot.com/2011/07/fast-image-cloning-library-ive-written.html http://methodart.blogspot.com/2011/07/fast-image-cloning-lib...
chintan 15y ago
Brilliant!!
tnc 15y ago
Here's the paper: http://www.cs.huji.ac.il/~danix/mvclone/ http://www.cs.huji.ac.il/~danix/mvclone/
dadads 15y ago
I approve of this!
AdamTReineke 15y ago
High-res source 3D face scans + a Kinect to track target head position and rotation better would hide the modifications quite well. Awesome project.
andrewpi 15y ago
Reminds me somewhat of the scramble suit from A Scanner Darkly!
Geee 15y ago
Really awesome. That second video shows much better results. Now, still some fine-tuning and someone could create application where people can try out different make-ups or eyeglasses.
- rohit89 15y ago
  Hair styling is another big area where this would be really useful.
croddin 15y ago
If it is open source, where is the code? I am only seeing links to libraries it uses.
- grillz 15y ago
  Linked to from the Vimeo page: "FaceTracker is available for download (for research purposes only). The library includes the C/C++ API, example code for interfacing with the API, a pre-trained model and documentation. To download it, please send an email to Jason Saragih" http://web.mac.com/jsaragih/FaceTracker/FaceTracker.html http://web.mac.com/jsaragih/FaceTracker/FaceTracker.html
- mdda 15y ago
  The thing being claimed as Open Source is OpenFrameworks (https://github.com/openframeworks/openFrameworks/ https://github.com/openframeworks/openFrameworks/), which is MIT licensed. However the source for FaceTracker looks like it's "please email us" - so the licensing for that isn't clear. The end result could be (legitimately) proprietary, as long as they supply the MIT license notice.
cfontes 15y ago
Really cool stuff. would be nice to have this feature used together with augmented reality games. We could then use your preferred char outfit and face while playing a wii like game, so the game would present a video of you as the main char with any outfit, like playing streetfighter being Mario :D Really ingenious idea.
pavel_lishin 15y ago
How long before this becomes good enough to fool people on Skype? And how long before someone ends up writing software to detect this? I think this is the first time in my life that I've felt like I was living in a scifi novel.
- Aloisius 15y ago
  Real-time virtual puppeting has been done in movies/television and research for a while now and yes, it can easily fool people. A professor friend of mine Jeremy Bailenson at Stanford actually uses the Kinect to track facial movements and uses 3D models of others to create puppets in real-time. Even more interesting, he can morph your face with the person you're video conferencing with to create a feeling of commonality in them. He actually wrote a book on it called Infinite Reality [1] which talks about all kinds of ways people will probably get manipulated in the future. He talks about things like mirroring movements (which he can do automatically in a video conference), looking into the eyes of every participant in a group video conference and other really interesting psychological hacks. [1] http://www.amazon.com/Infinite-Reality-Avatars-Eternal-Revolution/dp/0061809500/ http://www.amazon.com/Infinite-Reality-Avatars-Eternal-Revol...
  - rglullis 15y ago
    It is amazing how DFW nails that in Infinite Jest. In the book, people have the technology to use video-phones, but they stopped using it after realizing that video-phones eliminate the convenience of being able to communicate without actually partaking in all other kinds of communication. A silly example: with a video phone, you must be paying attention at the person in the camera, can't talk to someone while clipping your nails. Also in the book, people started using more technology to cover for these problems, from software that would make you look "presentable" to a complete avatar that would simulate all the motions that one should do while on camera. In the end, people just realize that no one is actually using the video part of the phone anymore, so they just stop using it and go back to audio-only. So, as much as I think we will have the technology to fool someone as you mention, I don't think it will happen simply because no one will adopt that technology in the first place. Kottke also comments on it: http://kottke.org/10/06/david-foster-wallace-on-iphone-4s-facetime http://kottke.org/10/06/david-foster-wallace-on-iphone-4s-fa...
ericgearhart 15y ago
I think the "creepy" factor of the images is probably due to the "uncanny valley"... Pixar fought this effect when they were first rendering humans "The uncanny valley is a hypothesis in the field of robotics and 3D computer animation, which holds that when human replicas look and act almost, but not perfectly, like actual human beings, it causes a response of revulsion among human observers. The "valley" in question is a dip in a proposed graph of the positivity of human reaction as a function of a robot's human likeness." http://en.wikipedia.org/wiki/Uncanny_valley http://en.wikipedia.org/wiki/Uncanny_valley
- pavel_lishin 15y ago
  Once they get over that barrier, though, it'll become much creepier for an entirely different reason.
  - dolphenstein 15y ago
    The net will be awash with G.I.R.L. 2.0 !
- tsunamifury 15y ago
  This has nothing to do with the Uncanny Valley. This is creepy because masks add a deadening look to the face and we sense that. It seperates the person from the face.
  - DLWormwood 15y ago
    Read up on the "Uncanny Valley" again. Your example is one of the cases that lead to the development of the hypothesis in the first place. (The concept dates back to 1906 with a German paper.)
- mirkules 15y ago
  Interestingly, one of the hardest aspects of human rendering is hands. Hands flex, bend and the skin stretches in ways which are difficult to represent mathematically, which is why in the late 90s and early 2000s (when human rendering started to emerge), most rendered human beings always wore gloves or were hidden from view entirely.
cypherpunks01 15y ago
Cool! The title should be edited to describe the link though. And as for being an open source project, link/source please?
- wanorris 15y ago
  Here's at least some of the author's code. Not sure if it's the whole thing. https://github.com/kylemcdonald/ofxFaceTracker https://github.com/kylemcdonald/ofxFaceTracker Edit: that code doesn't actually cover the substitution. Here's more info: http://vimeo.com/29279198 http://vimeo.com/29279198
swah 15y ago
This would be much better than face pixelation on Google street view.
swah 15y ago
And once more, while we were discussing about a new language to replace javascript, some folk wrote a kickass software program in C++. :)
jewel 15y ago
I imagine this will do to movies what autotune did to music. In other words, you no longer will have to find someone who is both good at acting and attractive.
- ippisl 15y ago
  It could also let movie producers get rid of the high salaries paid for popular movie stars.
  - 5hoom 15y ago
    That's an interesting point, and one that I could easily imagine happening. I suppose the only problem for the movie producer is who holds copyright on a persons face? Imagine George Lucas & co. wants to make 'Indianna Jones 5 - the prequel' with this technology using a likeness of a young Harrison Ford. The studio owns the likeness of the character 'Indianna Jones', but surely Harrison Ford owns the likeness to his own face (IANAL, but curious if there's precedent for this sort of thing).
    - ippisl 15y ago
      I think the more common use case would be getting face copyright from someone beautifull , and building it to stardom. I think if a major studio would want to build stars that way , it won't take them long.max a few years. so there won't be a long transition period anyways.
rhizome 15y ago
Why aren't the videos embedded simply from Vimeo? Why do I have to wait on "uploads.notempire.com"? Not to mention that it's not behaving very "empire"-like.
apu 15y ago
For face replacement in video, the state-of-the-art is this upcoming SIGGRAPH Asia 2011 work: http://www.eecs.harvard.edu/~dale/docs/faceReplace_sa2011.mp4 http://www.eecs.harvard.edu/~dale/docs/faceReplace_sa2011.mp... http://www.eecs.harvard.edu/~dale/docs/faceReplace_sa2011.pdf http://www.eecs.harvard.edu/~dale/docs/faceReplace_sa2011.pd... Having worked on this problem before, I know how tough it is to escape the uncanny valley when doing replacement, and these guys have really done impressively well at it (albeit with a fair amount of manual preprocessing and in controlled situations).
Aqwis 15y ago
How do professional movie productions do this? For example the Winklevoss brothers in The Social Network both had the face of one of the actors.
- Geee 15y ago
  This is simple 're-texturing'. I.e. the image of other face is projected onto your face (or the 3D reconstruction of it). To actually change, re-render and replace the original geometry is much much more complicated, and also involves re-creating all direct and indirect illumination, which also needs full 3D reconstruction of the surroundings and light sources. That's how it's done in movies and it involves motion capture and huge amount of manual labor. Edit: Speaking of replacing with a generated 3D face. Winklewoss replacement could be done 'simply' by double filming both actors in same position and then photoshopping the faces replacing the one with another, frame-by-frame. Face-tracking could be used to align the faces.
- keane 15y ago
  Armie Hammer's face was digitally transplanted on top of Josh Pence's body in shots where both brothers appeared in the same shot during post. Otherwise, they would change angles and re-film a scene with Hammer as both. For a video that shows how they did this, see http://videos.nymag.com/video/Vulture-Exclusive-The-Winklevii http://videos.nymag.com/video/Vulture-Exclusive-The-Winklevi...
keane 15y ago
Made with: 1. OpenCV 2.0 - C++, C, Python interfaces; BSD license - http://opencv.willowgarage.com/wiki/ http://opencv.willowgarage.com/wiki/ 2. FaceTracker - C/C++ API; "research purposes only", to download email Jason.saragih@csiro.au - http://web.mac.com/jsaragih/FaceTracker/FaceTracker.html http://web.mac.com/jsaragih/FaceTracker/FaceTracker.html 3. Method Art Image Clone - realtime cloning library (from gts, glew, glib); MIT license - http://methodart.blogspot.com/2011/07/fast-image-cloning-library-ive-written.html http://methodart.blogspot.com/2011/07/fast-image-cloning-lib... 4. openFrameworks - C++ toolkit; MIT license - https://github.com/openframeworks/openFrameworks https://github.com/openframeworks/openFrameworks 5. FaceOSC (ofxFaceTracker) - active appearance model addon for openFrameworks; "open source" - https://github.com/kylemcdonald/ofxFaceTracker https://github.com/kylemcdonald/ofxFaceTracker
- kragen 15y ago
  So it might be open-source, but depends on the non-open-source (and not even publicly available) project FaceTracker?
  - keane 15y ago
    Kyle McDonald discusses this: "I imagine FaceOSC being used to prototype ideas surrounding face-based interaction. I created it because Jason Saragih, the researcher behind FaceTracker, uses an open source non-commercial license for his code. He asks that anyone who wants to use the code email him directly, as a way to keep track of the usage. This is great, but I know that one of the fastest ways to get cool stuff happening is to make new tools and research accessible to a wide audience. So I asked him if it would be ok to make a standalone app for people to prototype their ideas — even if they don't have access to the code. Everyone already 'speaks' OSC so I thought this would be the easiest way to get the technology out there. Eventually, if people need to integrate it into a single application, they can contact Jason directly and use my ofxFaceTracker addon to get started: https://github.com/kylemcdonald/ofxFaceTracker https://github.com/kylemcdonald/ofxFaceTracker And if they need to go the commercial route, there’s FaceAPI http://www.seeingmachines.com/product/faceapi/ http://www.seeingmachines.com/product/faceapi/ " Source: http://electronica.fm/news/110714/music-your-face-artist-kyle-mcdonald-talks-face-tracking-music-making-kinect http://electronica.fm/news/110714/music-your-face-artist-kyl...
deleted 15y ago
[deleted]
mcantor 15y ago
Wow, how uncannily timely. Just yesterday, I read the part of a speculative fiction near-future novel, "Halting States" by Charles Stross, which includes this as an interesting detail with an eye towards technical imperfection; one of the protagonists is bemused by someone's neck glitching up into their face during a video call while they use this sort of software.
apitaru 15y ago
Kyle just posted a new demo video - he's playing around with the idea of the "Scrambler Suite" from A Scanner Darkly - http://vimeo.com/29391633 http://vimeo.com/29391633
- bh42222 15y ago
  Very cool. Much better than the effect used in the movie.
protagonist_h 15y ago
This could be used in video call centers in the future. Image you make a video call to your bank, and a blonde girl appears on the screen. In reality, however, you are talking to a dude in India. However, this would also require "voice substitution."
zerostar07 15y ago
This will be perfect for plastic surgeon and hairdressing applications. Also to try out a new smirk before actually growing it.
atomicdog 15y ago
Online dating just got a whole lot riskier...

456 points by monochromatic 15y ago

Here's where his broken link meant to go: http://methodart.blogspot.com/2011/07/fast-image-cloning-library-ive-written.html http://methodart.blogspot.com/2011/07/fast-image-cloning-lib...

chintan 15y ago

Brilliant!!

tnc 15y ago

Here's the paper: http://www.cs.huji.ac.il/~danix/mvclone/ http://www.cs.huji.ac.il/~danix/mvclone/

dadads 15y ago

I approve of this!

AdamTReineke 15y ago

High-res source 3D face scans + a Kinect to track target head position and rotation better would hide the modifications quite well. Awesome project.

andrewpi 15y ago

Reminds me somewhat of the scramble suit from A Scanner Darkly!

Geee 15y ago

Really awesome. That second video shows much better results. Now, still some fine-tuning and someone could create application where people can try out different make-ups or eyeglasses.

rohit89 15y ago
Hair styling is another big area where this would be really useful.

croddin 15y ago

If it is open source, where is the code? I am only seeing links to libraries it uses.

grillz 15y ago
Linked to from the Vimeo page: "FaceTracker is available for download (for research purposes only). The library includes the C/C++ API, example code for interfacing with the API, a pre-trained model and documentation. To download it, please send an email to Jason Saragih" http://web.mac.com/jsaragih/FaceTracker/FaceTracker.html http://web.mac.com/jsaragih/FaceTracker/FaceTracker.html
mdda 15y ago
The thing being claimed as Open Source is OpenFrameworks (https://github.com/openframeworks/openFrameworks/ https://github.com/openframeworks/openFrameworks/), which is MIT licensed. However the source for FaceTracker looks like it's "please email us" - so the licensing for that isn't clear. The end result could be (legitimately) proprietary, as long as they supply the MIT license notice.

cfontes 15y ago

Really cool stuff. would be nice to have this feature used together with augmented reality games. We could then use your preferred char outfit and face while playing a wii like game, so the game would present a video of you as the main char with any outfit, like playing streetfighter being Mario :D Really ingenious idea.

pavel_lishin 15y ago

How long before this becomes good enough to fool people on Skype? And how long before someone ends up writing software to detect this? I think this is the first time in my life that I've felt like I was living in a scifi novel.

Aloisius 15y ago
Real-time virtual puppeting has been done in movies/television and research for a while now and yes, it can easily fool people. A professor friend of mine Jeremy Bailenson at Stanford actually uses the Kinect to track facial movements and uses 3D models of others to create puppets in real-time. Even more interesting, he can morph your face with the person you're video conferencing with to create a feeling of commonality in them. He actually wrote a book on it called Infinite Reality [1] which talks about all kinds of ways people will probably get manipulated in the future. He talks about things like mirroring movements (which he can do automatically in a video conference), looking into the eyes of every participant in a group video conference and other really interesting psychological hacks. [1] http://www.amazon.com/Infinite-Reality-Avatars-Eternal-Revolution/dp/0061809500/ http://www.amazon.com/Infinite-Reality-Avatars-Eternal-Revol...
- rglullis 15y ago
  It is amazing how DFW nails that in Infinite Jest. In the book, people have the technology to use video-phones, but they stopped using it after realizing that video-phones eliminate the convenience of being able to communicate without actually partaking in all other kinds of communication. A silly example: with a video phone, you must be paying attention at the person in the camera, can't talk to someone while clipping your nails. Also in the book, people started using more technology to cover for these problems, from software that would make you look "presentable" to a complete avatar that would simulate all the motions that one should do while on camera. In the end, people just realize that no one is actually using the video part of the phone anymore, so they just stop using it and go back to audio-only. So, as much as I think we will have the technology to fool someone as you mention, I don't think it will happen simply because no one will adopt that technology in the first place. Kottke also comments on it: http://kottke.org/10/06/david-foster-wallace-on-iphone-4s-facetime http://kottke.org/10/06/david-foster-wallace-on-iphone-4s-fa...

ericgearhart 15y ago

I think the "creepy" factor of the images is probably due to the "uncanny valley"... Pixar fought this effect when they were first rendering humans "The uncanny valley is a hypothesis in the field of robotics and 3D computer animation, which holds that when human replicas look and act almost, but not perfectly, like actual human beings, it causes a response of revulsion among human observers. The "valley" in question is a dip in a proposed graph of the positivity of human reaction as a function of a robot's human likeness." http://en.wikipedia.org/wiki/Uncanny_valley http://en.wikipedia.org/wiki/Uncanny_valley

pavel_lishin 15y ago
Once they get over that barrier, though, it'll become much creepier for an entirely different reason.
- dolphenstein 15y ago
  The net will be awash with G.I.R.L. 2.0 !
tsunamifury 15y ago
This has nothing to do with the Uncanny Valley. This is creepy because masks add a deadening look to the face and we sense that. It seperates the person from the face.
- DLWormwood 15y ago
  Read up on the "Uncanny Valley" again. Your example is one of the cases that lead to the development of the hypothesis in the first place. (The concept dates back to 1906 with a German paper.)
mirkules 15y ago
Interestingly, one of the hardest aspects of human rendering is hands. Hands flex, bend and the skin stretches in ways which are difficult to represent mathematically, which is why in the late 90s and early 2000s (when human rendering started to emerge), most rendered human beings always wore gloves or were hidden from view entirely.

cypherpunks01 15y ago

Cool! The title should be edited to describe the link though. And as for being an open source project, link/source please?

wanorris 15y ago
Here's at least some of the author's code. Not sure if it's the whole thing. https://github.com/kylemcdonald/ofxFaceTracker https://github.com/kylemcdonald/ofxFaceTracker Edit: that code doesn't actually cover the substitution. Here's more info: http://vimeo.com/29279198 http://vimeo.com/29279198

swah 15y ago

This would be much better than face pixelation on Google street view.

swah 15y ago

And once more, while we were discussing about a new language to replace javascript, some folk wrote a kickass software program in C++. :)

jewel 15y ago

I imagine this will do to movies what autotune did to music. In other words, you no longer will have to find someone who is both good at acting and attractive.

ippisl 15y ago
It could also let movie producers get rid of the high salaries paid for popular movie stars.
- 5hoom 15y ago
  That's an interesting point, and one that I could easily imagine happening. I suppose the only problem for the movie producer is who holds copyright on a persons face? Imagine George Lucas & co. wants to make 'Indianna Jones 5 - the prequel' with this technology using a likeness of a young Harrison Ford. The studio owns the likeness of the character 'Indianna Jones', but surely Harrison Ford owns the likeness to his own face (IANAL, but curious if there's precedent for this sort of thing).
  - ippisl 15y ago
    I think the more common use case would be getting face copyright from someone beautifull , and building it to stardom. I think if a major studio would want to build stars that way , it won't take them long.max a few years. so there won't be a long transition period anyways.

rhizome 15y ago

Why aren't the videos embedded simply from Vimeo? Why do I have to wait on "uploads.notempire.com"? Not to mention that it's not behaving very "empire"-like.

apu 15y ago

For face replacement in video, the state-of-the-art is this upcoming SIGGRAPH Asia 2011 work: http://www.eecs.harvard.edu/~dale/docs/faceReplace_sa2011.mp4 http://www.eecs.harvard.edu/~dale/docs/faceReplace_sa2011.mp... http://www.eecs.harvard.edu/~dale/docs/faceReplace_sa2011.pdf http://www.eecs.harvard.edu/~dale/docs/faceReplace_sa2011.pd... Having worked on this problem before, I know how tough it is to escape the uncanny valley when doing replacement, and these guys have really done impressively well at it (albeit with a fair amount of manual preprocessing and in controlled situations).

Aqwis 15y ago

How do professional movie productions do this? For example the Winklevoss brothers in The Social Network both had the face of one of the actors.

Geee 15y ago
This is simple 're-texturing'. I.e. the image of other face is projected onto your face (or the 3D reconstruction of it). To actually change, re-render and replace the original geometry is much much more complicated, and also involves re-creating all direct and indirect illumination, which also needs full 3D reconstruction of the surroundings and light sources. That's how it's done in movies and it involves motion capture and huge amount of manual labor. Edit: Speaking of replacing with a generated 3D face. Winklewoss replacement could be done 'simply' by double filming both actors in same position and then photoshopping the faces replacing the one with another, frame-by-frame. Face-tracking could be used to align the faces.
keane 15y ago
Armie Hammer's face was digitally transplanted on top of Josh Pence's body in shots where both brothers appeared in the same shot during post. Otherwise, they would change angles and re-film a scene with Hammer as both. For a video that shows how they did this, see http://videos.nymag.com/video/Vulture-Exclusive-The-Winklevii http://videos.nymag.com/video/Vulture-Exclusive-The-Winklevi...

keane 15y ago

Made with: 1. OpenCV 2.0 - C++, C, Python interfaces; BSD license - http://opencv.willowgarage.com/wiki/ http://opencv.willowgarage.com/wiki/ 2. FaceTracker - C/C++ API; "research purposes only", to download email Jason.saragih@csiro.au - http://web.mac.com/jsaragih/FaceTracker/FaceTracker.html http://web.mac.com/jsaragih/FaceTracker/FaceTracker.html 3. Method Art Image Clone - realtime cloning library (from gts, glew, glib); MIT license - http://methodart.blogspot.com/2011/07/fast-image-cloning-library-ive-written.html http://methodart.blogspot.com/2011/07/fast-image-cloning-lib... 4. openFrameworks - C++ toolkit; MIT license - https://github.com/openframeworks/openFrameworks https://github.com/openframeworks/openFrameworks 5. FaceOSC (ofxFaceTracker) - active appearance model addon for openFrameworks; "open source" - https://github.com/kylemcdonald/ofxFaceTracker https://github.com/kylemcdonald/ofxFaceTracker

kragen 15y ago
So it might be open-source, but depends on the non-open-source (and not even publicly available) project FaceTracker?
- keane 15y ago
  Kyle McDonald discusses this: "I imagine FaceOSC being used to prototype ideas surrounding face-based interaction. I created it because Jason Saragih, the researcher behind FaceTracker, uses an open source non-commercial license for his code. He asks that anyone who wants to use the code email him directly, as a way to keep track of the usage. This is great, but I know that one of the fastest ways to get cool stuff happening is to make new tools and research accessible to a wide audience. So I asked him if it would be ok to make a standalone app for people to prototype their ideas — even if they don't have access to the code. Everyone already 'speaks' OSC so I thought this would be the easiest way to get the technology out there. Eventually, if people need to integrate it into a single application, they can contact Jason directly and use my ofxFaceTracker addon to get started: https://github.com/kylemcdonald/ofxFaceTracker https://github.com/kylemcdonald/ofxFaceTracker And if they need to go the commercial route, there’s FaceAPI http://www.seeingmachines.com/product/faceapi/ http://www.seeingmachines.com/product/faceapi/ " Source: http://electronica.fm/news/110714/music-your-face-artist-kyle-mcdonald-talks-face-tracking-music-making-kinect http://electronica.fm/news/110714/music-your-face-artist-kyl...

deleted 15y ago

[deleted]

mcantor 15y ago

Wow, how uncannily timely. Just yesterday, I read the part of a speculative fiction near-future novel, "Halting States" by Charles Stross, which includes this as an interesting detail with an eye towards technical imperfection; one of the protagonists is bemused by someone's neck glitching up into their face during a video call while they use this sort of software.

apitaru 15y ago

Kyle just posted a new demo video - he's playing around with the idea of the "Scrambler Suite" from A Scanner Darkly - http://vimeo.com/29391633 http://vimeo.com/29391633

bh42222 15y ago
Very cool. Much better than the effect used in the movie.

protagonist_h 15y ago

This could be used in video call centers in the future. Image you make a video call to your bank, and a blonde girl appears on the screen. In reality, however, you are talking to a dude in India. However, this would also require "voice substitution."

zerostar07 15y ago

This will be perfect for plastic surgeon and hairdressing applications. Also to try out a new smirk before actually growing it.

atomicdog 15y ago

Online dating just got a whole lot riskier...