4 ms·

Zero Shot Object Tracking

83 points by jonbaer 5y ago

forgingahead 5y ago
This is cool, though I wonder how much more accurate this is compared to dlib's built-in Correlation Tracker: http://dlib.net/correlation_tracker.py.html http://dlib.net/correlation_tracker.py.html Problem areas are usually when objects cross over each other, and when the tracked object starts moving at a faster speed (eg: sudden speed up in runners, or so on). *Edit: There's a good gif of dlib's implementation here: https://www.pyimagesearch.com/2018/10/22/object-tracking-with-dlib/ https://www.pyimagesearch.com/2018/10/22/object-tracking-wit...
- yeldarb 5y ago
  We haven’t had a chance to run it through eval on standard datasets yet to see how it compares quantitatively. I’d like to compare it to some of these: https://paperswithcode.com/task/multi-object-tracking https://paperswithcode.com/task/multi-object-tracking The code is available here if anyone wants to give it a go before we can get to it: https://github.com/roboflow-ai/zero-shot-object-tracking https://github.com/roboflow-ai/zero-shot-object-tracking
- nl 5y ago
  I believe DLib's tracker is based on tracking hand-made features - probably Cmashift or similar (similar to HOG features on their non-CNN face detector). I'd note that it hasn't been updated sine 2018 as well. That means the accuracy will be a lot lower - although it will be fast. But the Roboflow tracker is using OpenAI Clip, which seems a complete overkill and will be pretty slow! Something using a lightweight object detector should be much faster. This post shows how to use YOLO, which is the first things I'd try would be: https://github.com/yehengchen/Object-Detection-and-Tracking https://github.com/yehengchen/Object-Detection-and-Tracking
- jotaf 5y ago
  There's also Siamese Networks, they're pretty fast and simple, and people keep updating their backbone CNNs to more modern (computationally expensive) ones.
jeeeb 5y ago
This looks interesting although even in their sample video you can see a lot of tracking loss and reidentification events. I do wonder how it would perform in more difficult scenarios with problems like partial obstruction, crowding and changing object scale (e.g. objects moving away from the camera). I implemented a production object tracking (using tracking by detection) system and these were the main challenges I encountered. Still a very interesting looking result though:)
lucb1e 5y ago
I know what object tracking is, but how can you do it with zero shots of the object? Curious what this name refers to, but it's quite buried. From the blog post, click through to the repository, then click in the readme on OpenAI's CLIP zero-shot image classifier to find: > "Zero-shot learning" is when a model attempts to predict a class it saw zero times in the training data. So, using a model trained on exclusively cats and dogs to then detect raccoons. Wikipedia on ZSL gives this example: > For example, given a set of images of animals to be classified, along with auxiliary textual descriptions of what animals look like, an AI which has been trained to recognize horses, but has never seen a zebra, can still recognize a zebra if it also knows that zebras look like striped horses. Coming back to the blog post with this understanding, this part is interesting: > The breakthrough in our zero shot object tracking repository is to use generalized CLIP object features, eliminating the need for you to make additional object track annotations Where "CLIP is a neural network trained on a variety of (image, text) pairs." (From the first source again, so CLIP is a pre-trained model, not the algorithm that you need to make the model.) Looking a bit at the CLIP repository from openai: > CLIP matches the performance of the original ResNet50 on ImageNet “zero-shot” without using any of the original 1.28M labeled examples Here it says they don't use any labeled examples? So it just knows that a fish is called fish? This field is confusing and such a rabbit hole ^^
- yeldarb 5y ago
  We could have done a better job of explaining CLIP in this post, but we've written about it quite extensively before[1][2][3][4][5] if you're interested. I'm convinced CLIP is the most important advancement in machine learning in the past year; it's incredibly versatile. In a nutshell, CLIP is a multimodal model that combines knowledge of English-language concepts with semantic knowledge of images. It can just as easily distinguish between an image of a "cat" and a "dog" as it can between "an illustration of Deadpool pretending to be a bunny rabbit"[6] and "an underwater scene in the style of Vincent Van Gogh"[7] and any other concept you can come up with[8] (even though it has definitely never seen those things in its training data[9]). This is how these CLIP+VQGAN notebooks can create such a symphony of artistic renderings[10] (CLIP steers the GANs towards its interpretation of any English string that the artist can imagine). [1] ELI5 CLIP: https://blog.roboflow.com/clip-model-eli5-beginner-guide/ https://blog.roboflow.com/clip-model-eli5-beginner-guide/ [2] How to Try CLIP: https://blog.roboflow.com/how-to-use-openai-clip/ https://blog.roboflow.com/how-to-use-openai-clip/ [3] Content Moderation with CLIP: https://blog.roboflow.com/zero-shot-content-moderation-openai-new-clip-model/ https://blog.roboflow.com/zero-shot-content-moderation-opena... [4] CLIP Prompt Engineering: https://blog.roboflow.com/openai-clip-prompt-engineering/ https://blog.roboflow.com/openai-clip-prompt-engineering/ [5] CLIP for Semantic Image Similarity: https://blog.roboflow.com/apples-csam-neuralhash-collision/ https://blog.roboflow.com/apples-csam-neuralhash-collision/ [6] Deadpool Pretending to be a Bunny Rabbit: https://paint.wtf/ranking/erFA7/uCzEnPBtKkgoBAhBesQY https://paint.wtf/ranking/erFA7/uCzEnPBtKkgoBAhBesQY [7] Underwater Van Gogh: https://paint.wtf/ranking/xboNk/0fjxp0kEi0WeDgkP3LZX https://paint.wtf/ranking/xboNk/0fjxp0kEi0WeDgkP3LZX [8] Other human drawings as judged by CLIP: https://paint.wtf/leaderboard https://paint.wtf/leaderboard [9] CLIP-judged Pictionary game: https://blog.roboflow.com/how-we-built-paint-wtf-an-ai-that-judges-your-art-100-000-submissions/ https://blog.roboflow.com/how-we-built-paint-wtf-an-ai-that-... [10] AI Generated Art with CLIP+VQGAN: https://blog.roboflow.com/ai-generated-art/ https://blog.roboflow.com/ai-generated-art/

83 points by jonbaer 5y ago

This is cool, though I wonder how much more accurate this is compared to dlib's built-in Correlation Tracker: http://dlib.net/correlation_tracker.py.html http://dlib.net/correlation_tracker.py.html Problem areas are usually when objects cross over each other, and when the tracked object starts moving at a faster speed (eg: sudden speed up in runners, or so on). *Edit: There's a good gif of dlib's implementation here: https://www.pyimagesearch.com/2018/10/22/object-tracking-with-dlib/ https://www.pyimagesearch.com/2018/10/22/object-tracking-wit...

yeldarb 5y ago
We haven’t had a chance to run it through eval on standard datasets yet to see how it compares quantitatively. I’d like to compare it to some of these: https://paperswithcode.com/task/multi-object-tracking https://paperswithcode.com/task/multi-object-tracking The code is available here if anyone wants to give it a go before we can get to it: https://github.com/roboflow-ai/zero-shot-object-tracking https://github.com/roboflow-ai/zero-shot-object-tracking
nl 5y ago
I believe DLib's tracker is based on tracking hand-made features - probably Cmashift or similar (similar to HOG features on their non-CNN face detector). I'd note that it hasn't been updated sine 2018 as well. That means the accuracy will be a lot lower - although it will be fast. But the Roboflow tracker is using OpenAI Clip, which seems a complete overkill and will be pretty slow! Something using a lightweight object detector should be much faster. This post shows how to use YOLO, which is the first things I'd try would be: https://github.com/yehengchen/Object-Detection-and-Tracking https://github.com/yehengchen/Object-Detection-and-Tracking
jotaf 5y ago
There's also Siamese Networks, they're pretty fast and simple, and people keep updating their backbone CNNs to more modern (computationally expensive) ones.

jeeeb 5y ago

This looks interesting although even in their sample video you can see a lot of tracking loss and reidentification events. I do wonder how it would perform in more difficult scenarios with problems like partial obstruction, crowding and changing object scale (e.g. objects moving away from the camera). I implemented a production object tracking (using tracking by detection) system and these were the main challenges I encountered. Still a very interesting looking result though:)

lucb1e 5y ago

I know what object tracking is, but how can you do it with zero shots of the object? Curious what this name refers to, but it's quite buried. From the blog post, click through to the repository, then click in the readme on OpenAI's CLIP zero-shot image classifier to find: > "Zero-shot learning" is when a model attempts to predict a class it saw zero times in the training data. So, using a model trained on exclusively cats and dogs to then detect raccoons. Wikipedia on ZSL gives this example: > For example, given a set of images of animals to be classified, along with auxiliary textual descriptions of what animals look like, an AI which has been trained to recognize horses, but has never seen a zebra, can still recognize a zebra if it also knows that zebras look like striped horses. Coming back to the blog post with this understanding, this part is interesting: > The breakthrough in our zero shot object tracking repository is to use generalized CLIP object features, eliminating the need for you to make additional object track annotations Where "CLIP is a neural network trained on a variety of (image, text) pairs." (From the first source again, so CLIP is a pre-trained model, not the algorithm that you need to make the model.) Looking a bit at the CLIP repository from openai: > CLIP matches the performance of the original ResNet50 on ImageNet “zero-shot” without using any of the original 1.28M labeled examples Here it says they don't use any labeled examples? So it just knows that a fish is called fish? This field is confusing and such a rabbit hole ^^

yeldarb 5y ago
We could have done a better job of explaining CLIP in this post, but we've written about it quite extensively before[1][2][3][4][5] if you're interested. I'm convinced CLIP is the most important advancement in machine learning in the past year; it's incredibly versatile. In a nutshell, CLIP is a multimodal model that combines knowledge of English-language concepts with semantic knowledge of images. It can just as easily distinguish between an image of a "cat" and a "dog" as it can between "an illustration of Deadpool pretending to be a bunny rabbit"[6] and "an underwater scene in the style of Vincent Van Gogh"[7] and any other concept you can come up with[8] (even though it has definitely never seen those things in its training data[9]). This is how these CLIP+VQGAN notebooks can create such a symphony of artistic renderings[10] (CLIP steers the GANs towards its interpretation of any English string that the artist can imagine). [1] ELI5 CLIP: https://blog.roboflow.com/clip-model-eli5-beginner-guide/ https://blog.roboflow.com/clip-model-eli5-beginner-guide/ [2] How to Try CLIP: https://blog.roboflow.com/how-to-use-openai-clip/ https://blog.roboflow.com/how-to-use-openai-clip/ [3] Content Moderation with CLIP: https://blog.roboflow.com/zero-shot-content-moderation-openai-new-clip-model/ https://blog.roboflow.com/zero-shot-content-moderation-opena... [4] CLIP Prompt Engineering: https://blog.roboflow.com/openai-clip-prompt-engineering/ https://blog.roboflow.com/openai-clip-prompt-engineering/ [5] CLIP for Semantic Image Similarity: https://blog.roboflow.com/apples-csam-neuralhash-collision/ https://blog.roboflow.com/apples-csam-neuralhash-collision/ [6] Deadpool Pretending to be a Bunny Rabbit: https://paint.wtf/ranking/erFA7/uCzEnPBtKkgoBAhBesQY https://paint.wtf/ranking/erFA7/uCzEnPBtKkgoBAhBesQY [7] Underwater Van Gogh: https://paint.wtf/ranking/xboNk/0fjxp0kEi0WeDgkP3LZX https://paint.wtf/ranking/xboNk/0fjxp0kEi0WeDgkP3LZX [8] Other human drawings as judged by CLIP: https://paint.wtf/leaderboard https://paint.wtf/leaderboard [9] CLIP-judged Pictionary game: https://blog.roboflow.com/how-we-built-paint-wtf-an-ai-that-judges-your-art-100-000-submissions/ https://blog.roboflow.com/how-we-built-paint-wtf-an-ai-that-... [10] AI Generated Art with CLIP+VQGAN: https://blog.roboflow.com/ai-generated-art/ https://blog.roboflow.com/ai-generated-art/