11 ms·

DensePose – Dense Human Pose Estimation in the Wild

298 points by calahad 8y ago

calahad 8y ago
This just popped up in my github feed, looks interesting.
icebraining 8y ago
Previous thread (I hadn't seen it either): https://news.ycombinator.com/item?id=16289057 https://news.ycombinator.com/item?id=16289057
jaxbot 8y ago
CC-Non commercial is a disappointing license to find. The project is very cool otherwise.
- nerdponx 8y ago
  Cheaper than a patent.
- black_puppydog 8y ago
  I wonder how they arrived at Creative Commons' CC-BY-NC as the license. These licenses are not meant for code but for artwork, Creative Commons actually discourage the use of their licenses for code [1]. I recently noticed the same with the FastPhotoStyle code [2] by nvidia, so I'm wondering if there is something that draws their legal departments to this license? [1]: https://creativecommons.org/faq/#can-i-apply-a-creative-commons-license-to-software https://creativecommons.org/faq/#can-i-apply-a-creative-comm... [2]: https://github.com/NVIDIA/FastPhotoStyle https://github.com/NVIDIA/FastPhotoStyle
  - detaro 8y ago
    discourage people that aren't just hobbyists messing around from using it?
    - black_puppydog 8y ago
      yeah, that's what I came up with myself. But I thought a main point, if not the whole point of publishing code for these companies was to appeal to developer-types who are fond of real open source/science. And those should be able to tell the difference... It's a bit like allowing your scientists to publish their research, but only in prohibitively expensive and thus exceedingly niche journals.
  - dr_zoidberg 8y ago
    Hmmm... I guess it's been selected that way because it'd covevr the model files and the datasets. It's not about the code as much as the datasets/models.
  - levesque 8y ago
    If the dataset it was trained on is CC-BY-NC, I'm pretty sure the model also has to be CC-BY-NC. However I think this is not respected, or even considered by most people. I'd go with limiting how competitors can use it as the main deciding factor.
    - m1sta_ 8y ago
      I'd never thought of this. It is a very interesting idea.
    - bigiain 8y ago
      <cynical thought>Note that two of the three team members are from Facebook...
  - rhizome 8y ago
    They want to be the only people who can sell this where the real money is: military and law enforcement.
- sequoia 8y ago
  Have you tried contacting the author/copyright holder to find out whether you can negotiate a commercial licensing agreement for the software? I assume you're open to paying commercial licensing fees.
bsenftner 8y ago
Makes me wonder how much better they have in closed source to release this openly...
- cdibona 8y ago
  It is closed, the non-commercial license means almost no one can use it.
natch 8y ago
I'm too much of a newbie to figure this out but maybe someone here can tell me: Do they provide the final trained model? Or just a precursor model and code that trains a final model given bring-your-own data?
- icebraining 8y ago
  The site says "The dataset will soon be available on this website!", but apparently it's been saying that for at least four months.
  - natch 8y ago
    Yes thanks. I was asking about the trained model though. I realize you can get the model by training with the data, but I don't believe for a minute that any data they release will be anything but a small fraction of what they trained their real model with.
- abrichr 8y ago
  Yes. From https://github.com/facebookresearch/DensePose/blob/master/GETTING_STARTED.md#inference-with-pretrained-models https://github.com/facebookresearch/DensePose/blob/master/GE...: > DensePose should automatically download the model from the URL specified by the --wts argument
bsenftner 8y ago
I find this highly interesting from a personal perspective, as I am an author of a Personalized Advertising patent[1] from the early 2000's. I was unable to capitalize on my work, and ultimately sold the patent. Interesting to note, the month the patents' expired (due to aggressive early filing dates of my original patents) I noticed Facebook's AR Kit was released, which touched multiple aspects of what is protected by the global patents. This tech is key for Personalized Advertising where consumers are inserted into still and video adverting in place of current spokespersons and side-by-side with celebrities. Advertising is about to get surreal and the fake news consumers are about to get exploited something unbelievable. "Deep Fakes" for porn is kid stuff compared to what this tech opens: Pandora's Box if you ask me. [1] https://patents.justia.com/inventor/blake-senftner https://patents.justia.com/inventor/blake-senftner
- searine 8y ago
  >This tech is key for Personalized Advertising where consumers are inserted into still and video adverting in place of current spokespersons and side-by-side with celebrities. What a boring dystopia. Seriously though, if a company ever did without my permission I would sue the pants off of them.
  - fabianhjr 8y ago
    The issue is that Facebook, Google et al. already get a ton of consent for targeted/personalized advertisement and usage of your images and data. So you might not see this in a mall, but you will surely see it online.
  - hbosch 8y ago
    By creating a Facebook account you should assume they have your permission.
    - icebraining 8y ago
      No, it's personal data, you need specific consent under the GDPR :)
  - 8y ago
simonvc 8y ago
No-one's said it yet so i will (but not my idea..) This is going to be super useful for a future Oculus AR headset. Basically, imagine the current oculus go headset, but with cameras on the front, and instead of showing you the actual world, it shows you a game, based on the existing current world, but morphed to look like Starship Troopers or something.
- matthewwiese 8y ago
  You could achieve similar results for cheaper by taking LSD. There's the added benefit that your brain actually believes it's all real instead of merely perceiving it.
  - joemi 8y ago
    Since you mentioned LSD, I wonder if technology like this could one day replace hallucinogenic drugs? At least partially - I'm sure some people will always prefer the chemical experience. I've always been pretty interested in the effects of hallucinogens, but could never bring myself to try any of them due the potential risks. I wonder how much "your brain actually believes it's all real instead of merely perceiving it" matters... I know there have been a few times where I've gotten so sucked into a movie in a darkened theater that when it ends its incredibly jarring to be brought back to the real world, and the times that's happened to me haven't even been 3D movies, let alone interactive like AR.
    - matthewwiese 8y ago
      What would be interesting is the effect of both together; a sort of Brave New World 2.0. Great fodder for the likes of Black Mirror, at least.
      - lioeters 8y ago
        Seeing how we live in an age of decadence and self-indulgence - perhaps just driven by instinct to pursue pleasure/curiosity - I'm sure that this is a possible future development: chemically-enhanced augmented reality experiences. "Augmented" in more than one sense of the word.
HealthGoth 8y ago
What's a dense human?
acd 8y ago
It can probably be used to identify potential criminals on the way the pose walk threatening pose. Police can then screen them. Like the movie Minority report.
- milesokeefe 8y ago
  Do you mean using gait analysis to identify humans and match them to a known criminal database or do you mean finding suspect criminals based on some “criminal” way they walk? If the latter, I don’t think that’s really based on anything more than current cultural profiling.
  - pavel_lishin 8y ago
    "Take that pep out of your step, citizen!"
  - sonnyblarney 8y ago
    " I don’t think that’s really based on anything more than current cultural profiling." Yes, but if those individuals were actually more likely to commit crime, the AI would learn those things anyhow, leaving us with the question: if a specific demographic is considerably more likely to commit crime, and the AI picks up on it, is the AI 'racist'? Because racism is a moral judgement, moreover, the intersectionalists would indicate that it also requires the notion of 'power'. This is not some novel issue I think and will fast become a real ethical dilemma.
  - acd 8y ago
    I mean by tracking criminals cell phones locations, watching how they walk. Checking the network effect ie if a person contacts a previous criminal in the same gang network. Matching photos form surveillance cameras and public social media pictures. Matching walking pose you can pretty much track down criminals. The power is when you combine the different databases and build a profile of the person. It is very similar to how advertisement companies like Google build up profiles of customers(gmail,search behavior,dns name resolution tracking,cookie tracking) only in a different field. It is probably even more powerful when you combine physical behavior with online behavior-
unit91 8y ago
A few years ago, I would have thought this technical feat was amazing, and stopped there. Now I think it's creepy. Ah how times have changed...
- cal5k 8y ago
  There's nothing inherently creepy about it. I imagine it will be very useful for adding context to a scene - which is important for self-driving cars, AR, image classification, etc. If you take a bunch of photos involving people doing things and extract pose information, I'd imagine it would be helpful in figuring out what's going on in other situations that are otherwise dissimilar.
- rojobuffalo 8y ago
  Is there any other upside to this besides novelty and amusement? It's easy to imagine a dozen applications where this would threaten human privacy and safety.
  - TeMPOraL 8y ago
    What's wrong with amusement? Seriously, I'd love to have this in videogames - to import myself and other people into the game. I'd also love to get my hands on a good, fully automated [series of photos] -> [textured 3D human model] pipeline, for anything from silly renders to 3D printing mementos. Technology is inherently usable both for good and evil. You get both by default. It takes active countermeasures - usually non-tech, like regulations - to limit evil applications without sacrificing the good ones. As a society, we do that to some extent, but unfortunately we're not as successful as I'd like (e.g. if it were up to me, I'd seriously curtail the advertising industry).
    - namibj 8y ago
      The structure from motion stuff is good enough if you got a couple of pictures from around the human without the human moving (much), but I suspect getting enough pictures without the human moving too far might be difficult.
    - 8y ago
ambicapter 8y ago
Is there any reason this would be restricted to human pose estimation, as opposed to, say, rangefinding of moving objects of a specific roughly-known size?
davidw 8y ago
There are a variety of technical things where I think "wow, that's pretty clever and/or innovative - hats off!", and then there are things like this where I'm like "OMG wizardry!". But that's one of the things I love about the field of "computer stuff": there's so much interesting stuff I don't know.
lunch 8y ago
I can imagine this tech being using in some pretty interesting/scary ways: * Generating avatars in Facebook's VR land from photos you're tagged in * Recognizing a person IRL from photos they're tagged in
deleted 8y ago
[deleted]
lwansbrough 8y ago
I can't tell - is this able to extrapolate pose information into three dimensions, or can it only project onto two dimensional scenes?
- gugagore 8y ago
  It's something in between the two possibilities you describe. Each human pixel in the image is labeled with an index and two coordinates: x, y (u and v are the traditional names, but think of 2D x, y coordinates) The index specifies which patch that pixel is on, and the x, y coordinates specify where in the patch the pixel is on. This is for a pre-specified set of patches that cover a human mesh. See https://github.com/facebookresearch/DensePose/blob/master/notebooks/DensePose-COCO-Visualize.ipynb https://github.com/facebookresearch/DensePose/blob/master/no... for more detail. So, no, it does not extrapolate the full mesh, but also for all human pixels, you are getting 3D information.
tastythrowaway 8y ago
So is this basically 3-D rotoscoping?
bigiain 8y ago
I _assume_ this would make it fairly easy to do bone-length estimation and comparison, leading to a way to uniquely identify someone from a video feed of them walking - even without any facial features... Now I wonder how much of that tech is already deployed...
- infectoid 8y ago
  Gait analysis has been around for awhile. I guess this technique just makes it even easier. What's crazy to think about is Gait Analysis from orbit https://www.schneier.com/blog/archives/2008/09/gait_analysis_f.html https://www.schneier.com/blog/archives/2008/09/gait_analysis...
ricardobeat 8y ago
Would it be possible to map the first detected pose to a 3D model, scale and deform it to match the pose, and then use each next pose to manipulate the 3D model (vs generating all the vertices again)? This should result in smooth animation, without artifacts, and joint limits might even help with position estimation.
tempodox 8y ago
Leave it to FB to come up with the most creepy and invasive tracking and surveillance.

298 points by calahad 8y ago

This just popped up in my github feed, looks interesting.

Previous thread (I hadn't seen it either): https://news.ycombinator.com/item?id=16289057 https://news.ycombinator.com/item?id=16289057

jaxbot 8y ago

CC-Non commercial is a disappointing license to find. The project is very cool otherwise.

nerdponx 8y ago
Cheaper than a patent.
black_puppydog 8y ago
I wonder how they arrived at Creative Commons' CC-BY-NC as the license. These licenses are not meant for code but for artwork, Creative Commons actually discourage the use of their licenses for code [1]. I recently noticed the same with the FastPhotoStyle code [2] by nvidia, so I'm wondering if there is something that draws their legal departments to this license? [1]: https://creativecommons.org/faq/#can-i-apply-a-creative-commons-license-to-software https://creativecommons.org/faq/#can-i-apply-a-creative-comm... [2]: https://github.com/NVIDIA/FastPhotoStyle https://github.com/NVIDIA/FastPhotoStyle
- detaro 8y ago
  discourage people that aren't just hobbyists messing around from using it?
  - black_puppydog 8y ago
    yeah, that's what I came up with myself. But I thought a main point, if not the whole point of publishing code for these companies was to appeal to developer-types who are fond of real open source/science. And those should be able to tell the difference... It's a bit like allowing your scientists to publish their research, but only in prohibitively expensive and thus exceedingly niche journals.
- dr_zoidberg 8y ago
  Hmmm... I guess it's been selected that way because it'd covevr the model files and the datasets. It's not about the code as much as the datasets/models.
- levesque 8y ago
  If the dataset it was trained on is CC-BY-NC, I'm pretty sure the model also has to be CC-BY-NC. However I think this is not respected, or even considered by most people. I'd go with limiting how competitors can use it as the main deciding factor.
  - m1sta_ 8y ago
    I'd never thought of this. It is a very interesting idea.
  - bigiain 8y ago
    <cynical thought>Note that two of the three team members are from Facebook...
- rhizome 8y ago
  They want to be the only people who can sell this where the real money is: military and law enforcement.
sequoia 8y ago
Have you tried contacting the author/copyright holder to find out whether you can negotiate a commercial licensing agreement for the software? I assume you're open to paying commercial licensing fees.

bsenftner 8y ago

Makes me wonder how much better they have in closed source to release this openly...

cdibona 8y ago
It is closed, the non-commercial license means almost no one can use it.

natch 8y ago

I'm too much of a newbie to figure this out but maybe someone here can tell me: Do they provide the final trained model? Or just a precursor model and code that trains a final model given bring-your-own data?

icebraining 8y ago
The site says "The dataset will soon be available on this website!", but apparently it's been saying that for at least four months.
- natch 8y ago
  Yes thanks. I was asking about the trained model though. I realize you can get the model by training with the data, but I don't believe for a minute that any data they release will be anything but a small fraction of what they trained their real model with.
abrichr 8y ago
Yes. From https://github.com/facebookresearch/DensePose/blob/master/GETTING_STARTED.md#inference-with-pretrained-models https://github.com/facebookresearch/DensePose/blob/master/GE...: > DensePose should automatically download the model from the URL specified by the --wts argument

bsenftner 8y ago

I find this highly interesting from a personal perspective, as I am an author of a Personalized Advertising patent[1] from the early 2000's. I was unable to capitalize on my work, and ultimately sold the patent. Interesting to note, the month the patents' expired (due to aggressive early filing dates of my original patents) I noticed Facebook's AR Kit was released, which touched multiple aspects of what is protected by the global patents. This tech is key for Personalized Advertising where consumers are inserted into still and video adverting in place of current spokespersons and side-by-side with celebrities. Advertising is about to get surreal and the fake news consumers are about to get exploited something unbelievable. "Deep Fakes" for porn is kid stuff compared to what this tech opens: Pandora's Box if you ask me. [1] https://patents.justia.com/inventor/blake-senftner https://patents.justia.com/inventor/blake-senftner

searine 8y ago
>This tech is key for Personalized Advertising where consumers are inserted into still and video adverting in place of current spokespersons and side-by-side with celebrities. What a boring dystopia. Seriously though, if a company ever did without my permission I would sue the pants off of them.
- fabianhjr 8y ago
  The issue is that Facebook, Google et al. already get a ton of consent for targeted/personalized advertisement and usage of your images and data. So you might not see this in a mall, but you will surely see it online.
- hbosch 8y ago
  By creating a Facebook account you should assume they have your permission.
  - icebraining 8y ago
    No, it's personal data, you need specific consent under the GDPR :)
- 8y ago

simonvc 8y ago

No-one's said it yet so i will (but not my idea..) This is going to be super useful for a future Oculus AR headset. Basically, imagine the current oculus go headset, but with cameras on the front, and instead of showing you the actual world, it shows you a game, based on the existing current world, but morphed to look like Starship Troopers or something.

matthewwiese 8y ago
You could achieve similar results for cheaper by taking LSD. There's the added benefit that your brain actually believes it's all real instead of merely perceiving it.
- joemi 8y ago
  Since you mentioned LSD, I wonder if technology like this could one day replace hallucinogenic drugs? At least partially - I'm sure some people will always prefer the chemical experience. I've always been pretty interested in the effects of hallucinogens, but could never bring myself to try any of them due the potential risks. I wonder how much "your brain actually believes it's all real instead of merely perceiving it" matters... I know there have been a few times where I've gotten so sucked into a movie in a darkened theater that when it ends its incredibly jarring to be brought back to the real world, and the times that's happened to me haven't even been 3D movies, let alone interactive like AR.
  - matthewwiese 8y ago
    What would be interesting is the effect of both together; a sort of Brave New World 2.0. Great fodder for the likes of Black Mirror, at least.
    - lioeters 8y ago
      Seeing how we live in an age of decadence and self-indulgence - perhaps just driven by instinct to pursue pleasure/curiosity - I'm sure that this is a possible future development: chemically-enhanced augmented reality experiences. "Augmented" in more than one sense of the word.

HealthGoth 8y ago

What's a dense human?

acd 8y ago

It can probably be used to identify potential criminals on the way the pose walk threatening pose. Police can then screen them. Like the movie Minority report.

milesokeefe 8y ago
Do you mean using gait analysis to identify humans and match them to a known criminal database or do you mean finding suspect criminals based on some “criminal” way they walk? If the latter, I don’t think that’s really based on anything more than current cultural profiling.
- pavel_lishin 8y ago
  "Take that pep out of your step, citizen!"
- sonnyblarney 8y ago
  " I don’t think that’s really based on anything more than current cultural profiling." Yes, but if those individuals were actually more likely to commit crime, the AI would learn those things anyhow, leaving us with the question: if a specific demographic is considerably more likely to commit crime, and the AI picks up on it, is the AI 'racist'? Because racism is a moral judgement, moreover, the intersectionalists would indicate that it also requires the notion of 'power'. This is not some novel issue I think and will fast become a real ethical dilemma.
- acd 8y ago
  I mean by tracking criminals cell phones locations, watching how they walk. Checking the network effect ie if a person contacts a previous criminal in the same gang network. Matching photos form surveillance cameras and public social media pictures. Matching walking pose you can pretty much track down criminals. The power is when you combine the different databases and build a profile of the person. It is very similar to how advertisement companies like Google build up profiles of customers(gmail,search behavior,dns name resolution tracking,cookie tracking) only in a different field. It is probably even more powerful when you combine physical behavior with online behavior-

unit91 8y ago

A few years ago, I would have thought this technical feat was amazing, and stopped there. Now I think it's creepy. Ah how times have changed...

cal5k 8y ago
There's nothing inherently creepy about it. I imagine it will be very useful for adding context to a scene - which is important for self-driving cars, AR, image classification, etc. If you take a bunch of photos involving people doing things and extract pose information, I'd imagine it would be helpful in figuring out what's going on in other situations that are otherwise dissimilar.
rojobuffalo 8y ago
Is there any other upside to this besides novelty and amusement? It's easy to imagine a dozen applications where this would threaten human privacy and safety.
- TeMPOraL 8y ago
  What's wrong with amusement? Seriously, I'd love to have this in videogames - to import myself and other people into the game. I'd also love to get my hands on a good, fully automated [series of photos] -> [textured 3D human model] pipeline, for anything from silly renders to 3D printing mementos. Technology is inherently usable both for good and evil. You get both by default. It takes active countermeasures - usually non-tech, like regulations - to limit evil applications without sacrificing the good ones. As a society, we do that to some extent, but unfortunately we're not as successful as I'd like (e.g. if it were up to me, I'd seriously curtail the advertising industry).
  - namibj 8y ago
    The structure from motion stuff is good enough if you got a couple of pictures from around the human without the human moving (much), but I suspect getting enough pictures without the human moving too far might be difficult.
  - 8y ago

ambicapter 8y ago

Is there any reason this would be restricted to human pose estimation, as opposed to, say, rangefinding of moving objects of a specific roughly-known size?

davidw 8y ago

There are a variety of technical things where I think "wow, that's pretty clever and/or innovative - hats off!", and then there are things like this where I'm like "OMG wizardry!". But that's one of the things I love about the field of "computer stuff": there's so much interesting stuff I don't know.

lunch 8y ago

I can imagine this tech being using in some pretty interesting/scary ways: * Generating avatars in Facebook's VR land from photos you're tagged in * Recognizing a person IRL from photos they're tagged in

deleted 8y ago

[deleted]

lwansbrough 8y ago

I can't tell - is this able to extrapolate pose information into three dimensions, or can it only project onto two dimensional scenes?

gugagore 8y ago
It's something in between the two possibilities you describe. Each human pixel in the image is labeled with an index and two coordinates: x, y (u and v are the traditional names, but think of 2D x, y coordinates) The index specifies which patch that pixel is on, and the x, y coordinates specify where in the patch the pixel is on. This is for a pre-specified set of patches that cover a human mesh. See https://github.com/facebookresearch/DensePose/blob/master/notebooks/DensePose-COCO-Visualize.ipynb https://github.com/facebookresearch/DensePose/blob/master/no... for more detail. So, no, it does not extrapolate the full mesh, but also for all human pixels, you are getting 3D information.

tastythrowaway 8y ago

So is this basically 3-D rotoscoping?

bigiain 8y ago

I _assume_ this would make it fairly easy to do bone-length estimation and comparison, leading to a way to uniquely identify someone from a video feed of them walking - even without any facial features... Now I wonder how much of that tech is already deployed...

infectoid 8y ago
Gait analysis has been around for awhile. I guess this technique just makes it even easier. What's crazy to think about is Gait Analysis from orbit https://www.schneier.com/blog/archives/2008/09/gait_analysis_f.html https://www.schneier.com/blog/archives/2008/09/gait_analysis...

ricardobeat 8y ago

Would it be possible to map the first detected pose to a 3D model, scale and deform it to match the pose, and then use each next pose to manipulate the 3D model (vs generating all the vertices again)? This should result in smooth animation, without artifacts, and joint limits might even help with position estimation.

tempodox 8y ago

Leave it to FB to come up with the most creepy and invasive tracking and surveillance.