15 ms·

Meta AI releases CoTracker, a model for tracking any points (pixels) on a video

345 points by crakenzak 3y ago

crakenzak 3y ago
Paper: https://arxiv.org/abs/2307.07635 https://arxiv.org/abs/2307.07635 Github: https://github.com/facebookresearch/co-tracker https://github.com/facebookresearch/co-tracker Demo: https://huggingface.co/spaces/facebook/cotracker https://huggingface.co/spaces/facebook/cotracker
- Pr0ject217 3y ago
  Demo has OOM error =(
  - capableweb 3y ago
    I think it's meant to run in a private huggingface space, it's trying to allocate 46GB of VRAM. Edit: Hm, seems to work fine now regardless
iFire 3y ago
LICENSE Attribution-NonCommercial 4.0 International https://github.com/facebookresearch/co-tracker/blob/main/LICENSE.md https://github.com/facebookresearch/co-tracker/blob/main/LIC...
- ninjin 3y ago
  This is sadly the standard license for work produced at Facebook AI Research (FAIR) going back at least three or so years. Most researchers are not deeply invested into the details of licensing, so many of them simply assume it is properly open when they see Creative Commons and move on to do more research. This of course ignores the obvious problems with non-commercial licenses, such as the fact that the definition of a commercial entity might very well involve large chunks of the academic world. Thus, the choice of whatever bloodsu^Wlawyers inside Facebook that pushed this through as the default has had very nasty consequences. However, researchers inside FAIR can apply internally to have an actually open license applied to their work and I have never heard of it being denied. It is not fun (paperwork), but once you sit down and calmly explain the problems with non-commercial licensing and how it likely limits the uptake of their work, many will start pushing for it. My advice is thus just to try to stay positive and talk to the researchers about this rather than try to shame them (not that I have seen this in this thread). It should also be stated that although this licensing is not "good enough", it is still better than "Open"AI, DeepMind, and many others that gladly just sit on all but a miniscule amount of the code, data, and models they produce.
  - neilv 3y ago
    Thank you for the great explanation and actionable suggestions. I have to say, some of Meta's past open source and recent open-ish AI model releases have been big wins for the company's image in my mind.
  - nl 3y ago
    Can't upvote this enough. > It should also be stated that although this licensing is not "good enough", it is still better than "Open"AI, DeepMind, and many others that gladly just sit on all but a miniscule amount of the code, data, and models they produce. This is true, but Google Brain has historically been pretty good too.
  - nomad_horse 3y ago
    I think all FAIR researchers and engineers are aware of the NC-CA license limitations. They still use it because releasing the code for an already approved paper under NC-CA is super easy (~ self-approving by clicking a few buttons) vs following a slow open-sourcing process needed for the MIT license (includes approvals from a sponsoring director or two, committing to support the code for at least a year, etc). Releasing under MIT can easily take a few weeks with each stage requiring finding someone responsible and chasing them across time-zones. The best practice seems to release under NC-CA and re-license it later under MIT. Maybe this will happen here, too.
- deleted 3y ago
  [deleted]
- krasin 3y ago
  Fortunately, "Tracking Everything Everywhere All at Once" ([1], [2]) is Apache 2.0 and it's state of art. 1. https://omnimotion.github.io/ https://omnimotion.github.io/ 2. https://github.com/qianqianwang68/omnimotion https://github.com/qianqianwang68/omnimotion
  - imjonse 3y ago
    I wonder how useful these models are going to be for "Tracking Everybody Everywhere All at Once".
    - grandma_tea 3y ago
      Is that a sincere question or are you being glib? These models are still expensive to run at scale and probably don't compete with the phone in your pocket for tracking accuracy over a long time.
- moonchrome 3y ago
  Hmm would this be compatible with GPL ? I think not but not sure. Anyway this seems like it would be killer for blender to integrate camera tracking.
  - lambertsimnel 3y ago
    > Hmm would this be compatible with GPL ? I think not but not sure. Not a lawyer, but I don't believe so. The Free Software Foundation say of CC-BY-NC "This license does not qualify as free, because there are restrictions on charging money for copies."[0] The same page says "A nonfree license is automatically incompatible with the GNU GPL."[1] [0] https://www.gnu.org/licenses/license-list.html#CC-BY-NC https://www.gnu.org/licenses/license-list.html#CC-BY-NC [1] https://www.gnu.org/licenses/license-list.html#NonFreeSoftwareLicenses https://www.gnu.org/licenses/license-list.html#NonFreeSoftwa...
xnx 3y ago
Neat. It's mentioned on Facebook's page, but here Google's version of point tracking: https://deepmind-tapir.github.io https://deepmind-tapir.github.io which is Apache-2.0 licensed.
thethimble 3y ago
Does anyone understand the business angle for meta here with these models? I still don’t understand why their research division exists and how it relates to their core business. I’m a huge admirer of their work but don’t understand the why.
- BoorishBears 3y ago
  Commoditizing your compliment (https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/ https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/) Meta runs on UGC (User Generated Content), so making it easy and cheap to make great content is in their best interests. If they can reduce the cost of making a Billboard 100 quality song to $0, or make it so anyone with a 1.3MP webcam can use AI to get MKBHD level quality streams going, they: - increase the amount of content being generated - increase the quality of that content - make the content cheaper to acquire/license (since there's dramatically more supply)
  - marcopicentini 3y ago
    I understand but Code LLama or CoTracker do not create content for Facebook/Instagram. Moreover, AI-generated content is not so engageable as a human-generated content. I don't think the purpose is to have more content at lower price. The theory makes sense, but it's not clear what its complement is?
    - Aerbil313 3y ago
      > AI-generated content is not so engageable as a human-generated content Just wait until social media companies start to train transformers on engagement metrics.
- bg24 3y ago
  My hypothesis is that they want to get the absolute mindshare and adoption and decide the right product/solutions along the way. Note that there are trends (web 1, web 2, mobile etc.) and the leaders tend to grow significantly. It is a mad rush time with the new generation of AI, and future products are going to be amazing. Meta cannot afford to have just 1 cash cow. Metaverse failed, but they have a real shot here. A few things I can think of (beyond social networks). - Enterprise push, starting with very high-end enterprises. Their license structure (free below X million users) allows it nicely. They can afford to have a small team and yet win big. Note that typically 80% of any enterprise products revenue comes from say, 10-20% of top customers. - Build a platform. Meta learned from Google, and then Apple debacle on how dependent the world is on platform. What would the new OS look like? - Weaken Google and Apple. Some indications on why they partnered with Microsoft - this also ties to the point above.
tobr 3y ago
Not surprised that this performs so well, considering Facebook’s long experience with tracking pixels.
- d-lisp 3y ago
  long experience with tracking.
  - Kiro 3y ago
    You made the joke worse.
    - d-lisp 3y ago
      Come on, I was at Molière's level of comedy !
- FeepingCreature 3y ago
  ( A "tracking pixel" is a 1x1 remote image embedded in an email (or website) whose main purpose is that loading the URL confirms that the email was opened. )
  - deleted 3y ago
    [deleted]
jeffreygoesto 3y ago
Nice to see that Andrew Zisserman made it into the AI age. He and Hartley were my multi-view heros back in the days... And Faugeras of course...
petargyurov 3y ago
I think Meta's goals are becoming clearer: they want to make VR unbeliavable. Judging by this and by SAM, they want an AI system than can understand the world around itself in real time.
- zmmmmm 3y ago
  People do seem to vastly under estimate the time span over which Zuckerberg thinks and the end state he is going for. His vision is for a Facebook-like ubiquity of the tech. He sees AI as the pathway there because it means he can remove so many of the hardware sensors he otherwise would need that will be barriers to get the cost low enough. This is how he will make a little bit of money each from literally billions of users the way he has with Facebook. At this stage of development, he just needs these AI problems solved however that happens, whether it's internal or external. Releasing the tech only accelerates his broader version.
marcopicentini 3y ago
These open source AI model will create hundreds of AI startups with no competitive advantage. High competitive market and low margin.
- noiseinvacuum 3y ago
  Which is clearly a good thing for consumers and for startups who would otherwise have no chance of competing with tech behemoths like Google and MS.
runeks 3y ago
I wonder how this compares to the motion estimation algorithms in the x264 and x265 video codecs. If it's better, then it can be used to increase video compression, by using it at the motion estimation stage for these codecs.
throwaw12 3y ago
I wonder how research works inside Product companies. As an engineer working in a Product company, my focus switches between priorities set by PM and quarterly re-adjustments of the goals/strategies. Can't imagine same thing can be done in research: * Hey, when are we releasing model for tracking any points? * Can you estimate how long it takes to fix the issue you found in tracking accuracy?
- dan-robertson 3y ago
  Is Microsoft a Product company in your eyes? MSR has existed for a pretty long time and they seem to do a mix of self-directed research, publishing papers, etc, and talking to the other teams about the problems they have (eg excel getting FP or programming-by-example features had some MSR involvement. I don’t know how involved MSR were in stuff like some of the novel techniques used in bing or the RDMA-over-Ethernet things Microsoft tried to get to work in their datacentres (don’t recall if private or azure) – those things surely do count somewhat as research even if not MSR).
- riotnrrd 3y ago
  I work as a researcher (ML for robotics) inside a product company so I can speak to this a little. The research I work on generally takes a year between conception and publication (three months of tinkering, six months of refinement and benchmarking, three months to write and submit a paper). Now, I work on more than one of these projects at a time but they all have a similar cadence (conference deadlines do that for you), and can be "measured" by management the same way a year-long engineering project is. The questions I'm asked (or ask myself) and the same as when I was a pure engineer. Are you making headway? Are you on schedule? Do you need help? Should this project be abandoned or rescoped if it's not succeeding? Etc.
xwdv 3y ago
Oh wow, this will be the end of complicated motion capture rigs. Indie developers can do motion capture for their own complex 3D characters in the comfort of a home. Good.
ImHereToVote 3y ago
Are there models that can perform this in real-time? How does this stack up?
rasz 3y ago
15 years ago on single CPU core laptop https://www.robots.ox.ac.uk/~gk/PTAM/ https://www.robots.ox.ac.uk/~gk/PTAM/ Parallel Tracking and Mapping for Small AR Workspaces (PTAM) https://www.youtube.com/watch?v=Y9HMn6bd-v8 https://www.youtube.com/watch?v=Y9HMn6bd-v8 No ML, pure computer vision in C.
- grandma_tea 3y ago
  PTAM was a solution to a very different type of problem. Did you look at the videos? They are almost exclusively tracking points on soft, flexible bodies which is not possible with PTAM.
  - rasz 3y ago
    OK, 11 years ago then. This time with some very primitive ML: OpenTLD https://www.youtube.com/watch?v=1GhNXHCQGsM&t=228s https://www.youtube.com/watch?v=1GhNXHCQGsM&t=228s Was good enough for commercialization http://tldvision.com http://tldvision.com. I think their tech is powering this https://www.autosport.com/f1/news/ai-replays-and-more-augmented-reality-whats-new-for-f1s-tv-coverage-in-2023/10439342/ https://www.autosport.com/f1/news/ai-replays-and-more-augmen...
skenderbeu 3y ago
I'm sure there will be some future AR applicability with this
appplication 3y ago
One present modern challenge I’ve noticed is reverse video searching. There are no good platforms for this like there are reverse image. I wonder if this ability to quantize videos would allow you to build more efficient indices of videos that you could check against from some input.

345 points by crakenzak 3y ago

Paper: https://arxiv.org/abs/2307.07635 https://arxiv.org/abs/2307.07635 Github: https://github.com/facebookresearch/co-tracker https://github.com/facebookresearch/co-tracker Demo: https://huggingface.co/spaces/facebook/cotracker https://huggingface.co/spaces/facebook/cotracker

Pr0ject217 3y ago
Demo has OOM error =(
- capableweb 3y ago
  I think it's meant to run in a private huggingface space, it's trying to allocate 46GB of VRAM. Edit: Hm, seems to work fine now regardless

iFire 3y ago

LICENSE Attribution-NonCommercial 4.0 International https://github.com/facebookresearch/co-tracker/blob/main/LICENSE.md https://github.com/facebookresearch/co-tracker/blob/main/LIC...

ninjin 3y ago
This is sadly the standard license for work produced at Facebook AI Research (FAIR) going back at least three or so years. Most researchers are not deeply invested into the details of licensing, so many of them simply assume it is properly open when they see Creative Commons and move on to do more research. This of course ignores the obvious problems with non-commercial licenses, such as the fact that the definition of a commercial entity might very well involve large chunks of the academic world. Thus, the choice of whatever bloodsu^Wlawyers inside Facebook that pushed this through as the default has had very nasty consequences. However, researchers inside FAIR can apply internally to have an actually open license applied to their work and I have never heard of it being denied. It is not fun (paperwork), but once you sit down and calmly explain the problems with non-commercial licensing and how it likely limits the uptake of their work, many will start pushing for it. My advice is thus just to try to stay positive and talk to the researchers about this rather than try to shame them (not that I have seen this in this thread). It should also be stated that although this licensing is not "good enough", it is still better than "Open"AI, DeepMind, and many others that gladly just sit on all but a miniscule amount of the code, data, and models they produce.
- neilv 3y ago
  Thank you for the great explanation and actionable suggestions. I have to say, some of Meta's past open source and recent open-ish AI model releases have been big wins for the company's image in my mind.
- nl 3y ago
  Can't upvote this enough. > It should also be stated that although this licensing is not "good enough", it is still better than "Open"AI, DeepMind, and many others that gladly just sit on all but a miniscule amount of the code, data, and models they produce. This is true, but Google Brain has historically been pretty good too.
- nomad_horse 3y ago
  I think all FAIR researchers and engineers are aware of the NC-CA license limitations. They still use it because releasing the code for an already approved paper under NC-CA is super easy (~ self-approving by clicking a few buttons) vs following a slow open-sourcing process needed for the MIT license (includes approvals from a sponsoring director or two, committing to support the code for at least a year, etc). Releasing under MIT can easily take a few weeks with each stage requiring finding someone responsible and chasing them across time-zones. The best practice seems to release under NC-CA and re-license it later under MIT. Maybe this will happen here, too.
deleted 3y ago
[deleted]
krasin 3y ago
Fortunately, "Tracking Everything Everywhere All at Once" ([1], [2]) is Apache 2.0 and it's state of art. 1. https://omnimotion.github.io/ https://omnimotion.github.io/ 2. https://github.com/qianqianwang68/omnimotion https://github.com/qianqianwang68/omnimotion
- imjonse 3y ago
  I wonder how useful these models are going to be for "Tracking Everybody Everywhere All at Once".
  - grandma_tea 3y ago
    Is that a sincere question or are you being glib? These models are still expensive to run at scale and probably don't compete with the phone in your pocket for tracking accuracy over a long time.
moonchrome 3y ago
Hmm would this be compatible with GPL ? I think not but not sure. Anyway this seems like it would be killer for blender to integrate camera tracking.
- lambertsimnel 3y ago
  > Hmm would this be compatible with GPL ? I think not but not sure. Not a lawyer, but I don't believe so. The Free Software Foundation say of CC-BY-NC "This license does not qualify as free, because there are restrictions on charging money for copies."[0] The same page says "A nonfree license is automatically incompatible with the GNU GPL."[1] [0] https://www.gnu.org/licenses/license-list.html#CC-BY-NC https://www.gnu.org/licenses/license-list.html#CC-BY-NC [1] https://www.gnu.org/licenses/license-list.html#NonFreeSoftwareLicenses https://www.gnu.org/licenses/license-list.html#NonFreeSoftwa...

xnx 3y ago

Neat. It's mentioned on Facebook's page, but here Google's version of point tracking: https://deepmind-tapir.github.io https://deepmind-tapir.github.io which is Apache-2.0 licensed.

thethimble 3y ago

Does anyone understand the business angle for meta here with these models? I still don’t understand why their research division exists and how it relates to their core business. I’m a huge admirer of their work but don’t understand the why.

BoorishBears 3y ago
Commoditizing your compliment (https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/ https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/) Meta runs on UGC (User Generated Content), so making it easy and cheap to make great content is in their best interests. If they can reduce the cost of making a Billboard 100 quality song to $0, or make it so anyone with a 1.3MP webcam can use AI to get MKBHD level quality streams going, they: - increase the amount of content being generated - increase the quality of that content - make the content cheaper to acquire/license (since there's dramatically more supply)
- marcopicentini 3y ago
  I understand but Code LLama or CoTracker do not create content for Facebook/Instagram. Moreover, AI-generated content is not so engageable as a human-generated content. I don't think the purpose is to have more content at lower price. The theory makes sense, but it's not clear what its complement is?
  - Aerbil313 3y ago
    > AI-generated content is not so engageable as a human-generated content Just wait until social media companies start to train transformers on engagement metrics.
bg24 3y ago
My hypothesis is that they want to get the absolute mindshare and adoption and decide the right product/solutions along the way. Note that there are trends (web 1, web 2, mobile etc.) and the leaders tend to grow significantly. It is a mad rush time with the new generation of AI, and future products are going to be amazing. Meta cannot afford to have just 1 cash cow. Metaverse failed, but they have a real shot here. A few things I can think of (beyond social networks). - Enterprise push, starting with very high-end enterprises. Their license structure (free below X million users) allows it nicely. They can afford to have a small team and yet win big. Note that typically 80% of any enterprise products revenue comes from say, 10-20% of top customers. - Build a platform. Meta learned from Google, and then Apple debacle on how dependent the world is on platform. What would the new OS look like? - Weaken Google and Apple. Some indications on why they partnered with Microsoft - this also ties to the point above.

tobr 3y ago

Not surprised that this performs so well, considering Facebook’s long experience with tracking pixels.

d-lisp 3y ago
long experience with tracking.
- Kiro 3y ago
  You made the joke worse.
  - d-lisp 3y ago
    Come on, I was at Molière's level of comedy !
FeepingCreature 3y ago
( A "tracking pixel" is a 1x1 remote image embedded in an email (or website) whose main purpose is that loading the URL confirms that the email was opened. )
- deleted 3y ago
  [deleted]

jeffreygoesto 3y ago

Nice to see that Andrew Zisserman made it into the AI age. He and Hartley were my multi-view heros back in the days... And Faugeras of course...

petargyurov 3y ago

I think Meta's goals are becoming clearer: they want to make VR unbeliavable. Judging by this and by SAM, they want an AI system than can understand the world around itself in real time.

zmmmmm 3y ago
People do seem to vastly under estimate the time span over which Zuckerberg thinks and the end state he is going for. His vision is for a Facebook-like ubiquity of the tech. He sees AI as the pathway there because it means he can remove so many of the hardware sensors he otherwise would need that will be barriers to get the cost low enough. This is how he will make a little bit of money each from literally billions of users the way he has with Facebook. At this stage of development, he just needs these AI problems solved however that happens, whether it's internal or external. Releasing the tech only accelerates his broader version.

marcopicentini 3y ago

These open source AI model will create hundreds of AI startups with no competitive advantage. High competitive market and low margin.

noiseinvacuum 3y ago
Which is clearly a good thing for consumers and for startups who would otherwise have no chance of competing with tech behemoths like Google and MS.

runeks 3y ago

I wonder how this compares to the motion estimation algorithms in the x264 and x265 video codecs. If it's better, then it can be used to increase video compression, by using it at the motion estimation stage for these codecs.

throwaw12 3y ago

I wonder how research works inside Product companies. As an engineer working in a Product company, my focus switches between priorities set by PM and quarterly re-adjustments of the goals/strategies. Can't imagine same thing can be done in research: * Hey, when are we releasing model for tracking any points? * Can you estimate how long it takes to fix the issue you found in tracking accuracy?

dan-robertson 3y ago
Is Microsoft a Product company in your eyes? MSR has existed for a pretty long time and they seem to do a mix of self-directed research, publishing papers, etc, and talking to the other teams about the problems they have (eg excel getting FP or programming-by-example features had some MSR involvement. I don’t know how involved MSR were in stuff like some of the novel techniques used in bing or the RDMA-over-Ethernet things Microsoft tried to get to work in their datacentres (don’t recall if private or azure) – those things surely do count somewhat as research even if not MSR).
riotnrrd 3y ago
I work as a researcher (ML for robotics) inside a product company so I can speak to this a little. The research I work on generally takes a year between conception and publication (three months of tinkering, six months of refinement and benchmarking, three months to write and submit a paper). Now, I work on more than one of these projects at a time but they all have a similar cadence (conference deadlines do that for you), and can be "measured" by management the same way a year-long engineering project is. The questions I'm asked (or ask myself) and the same as when I was a pure engineer. Are you making headway? Are you on schedule? Do you need help? Should this project be abandoned or rescoped if it's not succeeding? Etc.

xwdv 3y ago

Oh wow, this will be the end of complicated motion capture rigs. Indie developers can do motion capture for their own complex 3D characters in the comfort of a home. Good.

ImHereToVote 3y ago

Are there models that can perform this in real-time? How does this stack up?

rasz 3y ago

15 years ago on single CPU core laptop https://www.robots.ox.ac.uk/~gk/PTAM/ https://www.robots.ox.ac.uk/~gk/PTAM/ Parallel Tracking and Mapping for Small AR Workspaces (PTAM) https://www.youtube.com/watch?v=Y9HMn6bd-v8 https://www.youtube.com/watch?v=Y9HMn6bd-v8 No ML, pure computer vision in C.

grandma_tea 3y ago
PTAM was a solution to a very different type of problem. Did you look at the videos? They are almost exclusively tracking points on soft, flexible bodies which is not possible with PTAM.
- rasz 3y ago
  OK, 11 years ago then. This time with some very primitive ML: OpenTLD https://www.youtube.com/watch?v=1GhNXHCQGsM&t=228s https://www.youtube.com/watch?v=1GhNXHCQGsM&t=228s Was good enough for commercialization http://tldvision.com http://tldvision.com. I think their tech is powering this https://www.autosport.com/f1/news/ai-replays-and-more-augmented-reality-whats-new-for-f1s-tv-coverage-in-2023/10439342/ https://www.autosport.com/f1/news/ai-replays-and-more-augmen...

skenderbeu 3y ago

I'm sure there will be some future AR applicability with this

appplication 3y ago

One present modern challenge I’ve noticed is reverse video searching. There are no good platforms for this like there are reverse image. I wonder if this ability to quantize videos would allow you to build more efficient indices of videos that you could check against from some input.