4 ms·

Vibe: Video Human Pose Estimation

93 points by ericsson_puma 6y ago

dbcurtis 6y ago
Restrictive license. The have a nice comparison to T-HMR, but I did not see a comparison to OpenPose. If that exists somewhere, please give me a pointer. [edit] OK, so I read a little further. Answered my own questions a bit, so here is what I found: VIBE produces 3D pose from 2D video. OpenPose needs stereo video for 3D pose estimation. OpenPose can produce 2D joint position tracking from 2D video. OpenPose 2D annotations were used to automate labeling input data sets for training VIBE. So, OpenPose: 2D video -> 2D pose estimate. VIBE 2D video -> 3D pose estimate.
- benrbray 6y ago
  You seem to point out the restrictive license (for non-commercial research purposes only) as if it's a negative, when I don't believe that it is. Why should companies be allowed to profit from a proprietary wrapper around open-source software built on top of decades of open-access research? Open research should stay open.
  - jchw 6y ago
    The main downside of any license that is restrictive is that it creates legal boundaries where no technical limitations exist, which on top of preventing the kinds of things you don’t want, can very well prevent totally desirable things as well (prime example is GPL compatibility with ZFS and Linux.) I am not really too concerned personally but suggesting a restrictive license has no downsides or there is not a valid viewpoint where a restrictive license is a negative doesn’t make sense, it’s just not true.
  - BorisTheBrave 6y ago
    It's not just non-commerical, it's for research purposes only!
    - AlleUndKalle 6y ago
      That is not true, their license allows almost everything other than commercial usage. Quote from their licence: "...To use the Model & Software for the sole purpose of performing non-commercial, scientific research, non-commercial education, or non-commercial artistic projects; Any other use, in particular any use for commercial purposes, is prohibited. This includes, without limitation, incorporation in a commercial product, use in a commercial service, or production of other artifacts for commercial purposes. The Model & Software may not be reproduced, modified and/or made available in any form to any third party without Max-Planck’s prior written permission."
      - qayxc 6y ago
        > their license allows almost everything other than commercial usage Nope. The license allows precisely three uses: research, education, and non-commercial art. That's it. In other words, even use in FOSS-compliant plugin's (e.g. for Blender) isn't allowed.
  - PostThisTooFast 6y ago
    Why should they not? It doesn't prevent anyone else from using the underlying technology. "Open research should stay open." How does someone building a product on it suddenly make it "not open?"
  - dbcurtis 6y ago
    > You seem to point out the restrictive license (for non-commercial research purposes only) as if it's a negative, Two word statement of fact: "Restrictive license." From those two words, you create a straw man. I have said before, here on HN and other places, that people should choose a license the way they choose a screwdriver, not the way they choose a religion. The license must fit the project goal. I point out the restrictive license in part because it is not one of the usual licenses that often accompanies source releases. A noteworthy oddity. An oddity that will cost money to understand, if need be. I point out the restrictive license in part to make people aware that they need to investigate before deciding to invest a lot of time evaluating this package. Simply a courtesy. As you point out, the license may or may not be a problem depending on the use case. I know of several entities that might be interested in taking a commercial license. I know of several entities that would have already stopped reading because the license does not fit their agenda.
- ericsson_puma 6y ago
  It also estimates the full body mesh, not only the 3D pose of a few joints.
SomeoneFromCA 6y ago
30 FPS on 2080 is not exactly fast
AlleUndKalle 6y ago
Colab demo is quite nice. These kind of research projects should definitely provide easy to use Colab notebooks for users to interact.
mandeepj 6y ago
I think it can be used in building a workout app to assess whether your pose is correct or not for the exercise you are doing
- dbcurtis 6y ago
  Indeed. Or other sports applications. I have a friend that is an elite-level track coach. She has an iPad app that helps analyze running form from video. This tech could be used to improve apps like that. My own little pet thought experiment for years has been an app that would track violin or 'cello bow motion to provide feedback to students on bowing technique. I am sure there are many similar applications.
- wellthisisgreat 6y ago
  occlusion is an impassable barrier in this case it seems. For some motions it can work, but if you can't see some limbs behind the back etc. there is not much that can be done regardless of the estimation
- mhh__ 6y ago
  Seems like a good use of the technology but this seems fairly computationally expensive
noodlesUK 6y ago
Are there any similar libraries to this or openpose that have FOSS compatible licenses?

93 points by ericsson_puma 6y ago

Restrictive license. The have a nice comparison to T-HMR, but I did not see a comparison to OpenPose. If that exists somewhere, please give me a pointer. [edit] OK, so I read a little further. Answered my own questions a bit, so here is what I found: VIBE produces 3D pose from 2D video. OpenPose needs stereo video for 3D pose estimation. OpenPose can produce 2D joint position tracking from 2D video. OpenPose 2D annotations were used to automate labeling input data sets for training VIBE. So, OpenPose: 2D video -> 2D pose estimate. VIBE 2D video -> 3D pose estimate.

benrbray 6y ago
You seem to point out the restrictive license (for non-commercial research purposes only) as if it's a negative, when I don't believe that it is. Why should companies be allowed to profit from a proprietary wrapper around open-source software built on top of decades of open-access research? Open research should stay open.
- jchw 6y ago
  The main downside of any license that is restrictive is that it creates legal boundaries where no technical limitations exist, which on top of preventing the kinds of things you don’t want, can very well prevent totally desirable things as well (prime example is GPL compatibility with ZFS and Linux.) I am not really too concerned personally but suggesting a restrictive license has no downsides or there is not a valid viewpoint where a restrictive license is a negative doesn’t make sense, it’s just not true.
- BorisTheBrave 6y ago
  It's not just non-commerical, it's for research purposes only!
  - AlleUndKalle 6y ago
    That is not true, their license allows almost everything other than commercial usage. Quote from their licence: "...To use the Model & Software for the sole purpose of performing non-commercial, scientific research, non-commercial education, or non-commercial artistic projects; Any other use, in particular any use for commercial purposes, is prohibited. This includes, without limitation, incorporation in a commercial product, use in a commercial service, or production of other artifacts for commercial purposes. The Model & Software may not be reproduced, modified and/or made available in any form to any third party without Max-Planck’s prior written permission."
    - qayxc 6y ago
      > their license allows almost everything other than commercial usage Nope. The license allows precisely three uses: research, education, and non-commercial art. That's it. In other words, even use in FOSS-compliant plugin's (e.g. for Blender) isn't allowed.
- PostThisTooFast 6y ago
  Why should they not? It doesn't prevent anyone else from using the underlying technology. "Open research should stay open." How does someone building a product on it suddenly make it "not open?"
- dbcurtis 6y ago
  > You seem to point out the restrictive license (for non-commercial research purposes only) as if it's a negative, Two word statement of fact: "Restrictive license." From those two words, you create a straw man. I have said before, here on HN and other places, that people should choose a license the way they choose a screwdriver, not the way they choose a religion. The license must fit the project goal. I point out the restrictive license in part because it is not one of the usual licenses that often accompanies source releases. A noteworthy oddity. An oddity that will cost money to understand, if need be. I point out the restrictive license in part to make people aware that they need to investigate before deciding to invest a lot of time evaluating this package. Simply a courtesy. As you point out, the license may or may not be a problem depending on the use case. I know of several entities that might be interested in taking a commercial license. I know of several entities that would have already stopped reading because the license does not fit their agenda.
ericsson_puma 6y ago
It also estimates the full body mesh, not only the 3D pose of a few joints.

SomeoneFromCA 6y ago

30 FPS on 2080 is not exactly fast

AlleUndKalle 6y ago

Colab demo is quite nice. These kind of research projects should definitely provide easy to use Colab notebooks for users to interact.

mandeepj 6y ago

I think it can be used in building a workout app to assess whether your pose is correct or not for the exercise you are doing

dbcurtis 6y ago
Indeed. Or other sports applications. I have a friend that is an elite-level track coach. She has an iPad app that helps analyze running form from video. This tech could be used to improve apps like that. My own little pet thought experiment for years has been an app that would track violin or 'cello bow motion to provide feedback to students on bowing technique. I am sure there are many similar applications.
wellthisisgreat 6y ago
occlusion is an impassable barrier in this case it seems. For some motions it can work, but if you can't see some limbs behind the back etc. there is not much that can be done regardless of the estimation
mhh__ 6y ago
Seems like a good use of the technology but this seems fairly computationally expensive

noodlesUK 6y ago

Are there any similar libraries to this or openpose that have FOSS compatible licenses?