6 ms·

Show HN: OpenCV based multi-object-tracker with various object detection models

105 points by adipandas 7y ago

binarysaurus 7y ago
There arw thousands of these repositories all over the place...
- adipandas 7y ago
  Okay, I am not sure about others. But I was working on this as a fun project. And I wanted to share my work. Hope you find it useful and please let me know your suggestions and any additional improvements I can make.
mendeza 7y ago
I work a lot with object detection at my current role. Would be cool to add Detectron 2, seems that this library that has some of the state of the art(SOTA) models in object detection. https://github.com/facebookresearch/detectron2 https://github.com/facebookresearch/detectron2 Also the TF Object Detection API has a suite of pretrained SOTA object detection models https://github.com/tensorflow/models/tree/master/research/object_detection https://github.com/tensorflow/models/tree/master/research/ob... What makes object detection really valuable is being able to finetune a model to detect objects that are relevant to your problem. However collecting, labeling, takes a lot of time since the best way is to manually label. But 30-50 hrs of manual labeling can get you some great results when you are leveraging pretrained models. Building a tool to manage data labeling and training for finetuning object detection models would be very valuable.
- mendeza 7y ago
  Also, what is a hard but an important question to figure out is how well are these models are preforming, and will they perform well to your problem. This paper gives a lot of insight into how image size and other factors really impact performance of modern object detection: https://arxiv.org/pdf/1611.10012.pdf https://arxiv.org/pdf/1611.10012.pdf. How large or small your object is in your dataset is also something to consider since there is a limit to how small of an object your detector can recognize. Also, I would recommend this repo to better understand how object detection models are evaluated as well: https://github.com/rafaelpadilla/Object-Detection-Metrics https://github.com/rafaelpadilla/Object-Detection-Metrics
- inetknght 7y ago
  > Building a tool to manage data labeling and training for finetuning object detection models would be very valuable. Something like recaptcha?
- adipandas 7y ago
  @mendeza, Thank you for the feedback. I am planning to add detectron models as well to this repository. Initial intention of this project was to avoid direct dependency of any deep learning framework. I have only used pre-trained models and opencv for this project. Making a tool for data labeling is an awesome idea. I will definitely like to work on it. We can collaborate on this if possible. You are also welcome to send in your pull requests.
  - mendeza 7y ago
    This is a good idea! TF Object Detection framework takes a lot of effort to use on simple ideas, so frameworks that enable more abstraction the better!
- ganeshkrishnan 7y ago
  Facebook has such awesome AI tools open sourced. I am more impressed with their code than Google
  - adipandas 7y ago
    Pytorch makes it even more fun.
- deppp 7y ago
  Check out our open-source tool called Label Studio, https://labelstud.io https://labelstud.io and we have a commercial offering that does exactly this, finetuning the models.
  - mendeza 7y ago
    I actually checked out your github repo and saw your framework that enables integration of trained models into your tool for active learning. Looks awesome! https://github.com/heartexlabs/pyheartex https://github.com/heartexlabs/pyheartex
  - mendeza 7y ago
    It would be great if your tool integrates popular object detection frameworks such as TF Object Detection API. Would save people a lot of headache installing and configuring in order to begin finetuning
martinshen 7y ago
Have you tried using object attributes/features to enable tracking instead of IOU-based tracking? I suspect this tracker falls apart when there are significant obstructions. Thoughts on a Recurrent YOLO or DeepSORT based tracker?
- adipandas 7y ago
  Yes. I am actually looking into this idea. I thought instead of recurrent yolo, Correlation Filter is much simpler. I was referring the following papers: 1. https://ethz.ch/content/dam/ethz/special-interest/baug/igp/photogrammetry-remote-sensing-dam/documents/pdf/learning-tracking-siamese.pdf https://ethz.ch/content/dam/ethz/special-interest/baug/igp/p... 2. https://zpascal.net/cvpr2018/He_A_Twofold_Siamese_CVPR_2018_paper.pdf https://zpascal.net/cvpr2018/He_A_Twofold_Siamese_CVPR_2018_... 3. http://openaccess.thecvf.com/content_ECCV_2018/papers/Yunhua_Zhang_Structured_Siamese_Network_ECCV_2018_paper.pdf http://openaccess.thecvf.com/content_ECCV_2018/papers/Yunhua...
- kevinsundar 7y ago
  I've found that IOU based tracking is surprisingly reliable (my dataset was primarily surveillance footage with a static camera) if you take measures to smooth the data and predict object motion using a kalman filter.
  - adipandas 7y ago
    Kalman filter is also one good way to reliably keep track of objects under significant obstruction.
nullbyte 7y ago
PyImageSearch is a fantastic resource for learning this stuff. Highly recommend it.
mbpqd 7y ago
Question for HN experts: what would be the best approach for object-detection when the object I am trying to detect is a specific image (not a class of objects) printed on a 2D surface (therefore it could be partially folded / curved, but still more of a distorted 2D object than a complex 3D one)?
- mendeza 7y ago
  I would check out template matching or using OpenCV feature extractors such as SIFT, SURF, ORB Template Matching scans an image with your template, smaller image and finds the location with highest cross correlation: https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_imgproc/py_template_matching/py_template_matching.html https://opencv-python-tutroals.readthedocs.io/en/latest/py_t... Template matching assumes the example image wont deform or take on different orientations. Popular Feature Extraction finds and derive more robust features that help find objects in the scenarios of distortion, orientation, and scale. Link for feature detection and matching: https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_feature2d/py_matcher/py_matcher.html https://opencv-python-tutroals.readthedocs.io/en/latest/py_t... Pyimagesearch does a cool method where they enable template matching to work at different scales. It ignores color information (by using edge detection), which could be a critical feature if the object has specific color: https://www.pyimagesearch.com/2015/01/26/multi-scale-template-matching-using-python-opencv/ https://www.pyimagesearch.com/2015/01/26/multi-scale-templat...
alexcnwy 7y ago
I've built several object detection systems and you definitely want to fine-tune your model to get good results, even if the object class you want to track is available in a pre-trained model (often it isn't). For example, pre-trained models don't work too well at detecting people in surveillance footage when the camera is mounted on the ceiling and people are at an angle in the footage. Off-the-shelf object detection models also don't work well in poor lighting conditions but can work well if you fine-tune one of the pre-trained models for your dataset and lighting conditions. Here's a great and fun tutorial on how to fine-tune an object detection model using TensorFlow (it's a surprisingly fiddly process but worth the headache): https://towardsdatascience.com/how-to-train-your-own-object-detector-with-tensorflows-object-detector-api-bec72ecfe1d9 https://towardsdatascience.com/how-to-train-your-own-object-... I suspect you can use your fine-tuned model with this repo by replacing the `get_ssd_model.sh` compiled graph with your fine-tuned model graph. It's also a fun exercise to implement the tracking yourself - here's a solid tutorial to get you started: https://www.pyimagesearch.com/2018/08/13/opencv-people-counter/ https://www.pyimagesearch.com/2018/08/13/opencv-people-count...
- adipandas 7y ago
  The top-down view problem is what I was facing when I was trying to use a detection model for a project with quadcopters. For this I had to train my object detector with the data we collected from that view. YOLO was used for this work. Yes, replacing the fine-tuned model with compiled graph should work. I tried to keep it simple and intuitive to change the model so as to allow the use of any other custom models.

105 points by adipandas 7y ago

binarysaurus 7y ago

There arw thousands of these repositories all over the place...

adipandas 7y ago
Okay, I am not sure about others. But I was working on this as a fun project. And I wanted to share my work. Hope you find it useful and please let me know your suggestions and any additional improvements I can make.

mendeza 7y ago

I work a lot with object detection at my current role. Would be cool to add Detectron 2, seems that this library that has some of the state of the art(SOTA) models in object detection. https://github.com/facebookresearch/detectron2 https://github.com/facebookresearch/detectron2 Also the TF Object Detection API has a suite of pretrained SOTA object detection models https://github.com/tensorflow/models/tree/master/research/object_detection https://github.com/tensorflow/models/tree/master/research/ob... What makes object detection really valuable is being able to finetune a model to detect objects that are relevant to your problem. However collecting, labeling, takes a lot of time since the best way is to manually label. But 30-50 hrs of manual labeling can get you some great results when you are leveraging pretrained models. Building a tool to manage data labeling and training for finetuning object detection models would be very valuable.

mendeza 7y ago
Also, what is a hard but an important question to figure out is how well are these models are preforming, and will they perform well to your problem. This paper gives a lot of insight into how image size and other factors really impact performance of modern object detection: https://arxiv.org/pdf/1611.10012.pdf https://arxiv.org/pdf/1611.10012.pdf. How large or small your object is in your dataset is also something to consider since there is a limit to how small of an object your detector can recognize. Also, I would recommend this repo to better understand how object detection models are evaluated as well: https://github.com/rafaelpadilla/Object-Detection-Metrics https://github.com/rafaelpadilla/Object-Detection-Metrics
inetknght 7y ago
> Building a tool to manage data labeling and training for finetuning object detection models would be very valuable. Something like recaptcha?
adipandas 7y ago
@mendeza, Thank you for the feedback. I am planning to add detectron models as well to this repository. Initial intention of this project was to avoid direct dependency of any deep learning framework. I have only used pre-trained models and opencv for this project. Making a tool for data labeling is an awesome idea. I will definitely like to work on it. We can collaborate on this if possible. You are also welcome to send in your pull requests.
- mendeza 7y ago
  This is a good idea! TF Object Detection framework takes a lot of effort to use on simple ideas, so frameworks that enable more abstraction the better!
ganeshkrishnan 7y ago
Facebook has such awesome AI tools open sourced. I am more impressed with their code than Google
- adipandas 7y ago
  Pytorch makes it even more fun.
deppp 7y ago
Check out our open-source tool called Label Studio, https://labelstud.io https://labelstud.io and we have a commercial offering that does exactly this, finetuning the models.
- mendeza 7y ago
  I actually checked out your github repo and saw your framework that enables integration of trained models into your tool for active learning. Looks awesome! https://github.com/heartexlabs/pyheartex https://github.com/heartexlabs/pyheartex
- mendeza 7y ago
  It would be great if your tool integrates popular object detection frameworks such as TF Object Detection API. Would save people a lot of headache installing and configuring in order to begin finetuning

martinshen 7y ago

Have you tried using object attributes/features to enable tracking instead of IOU-based tracking? I suspect this tracker falls apart when there are significant obstructions. Thoughts on a Recurrent YOLO or DeepSORT based tracker?

adipandas 7y ago
Yes. I am actually looking into this idea. I thought instead of recurrent yolo, Correlation Filter is much simpler. I was referring the following papers: 1. https://ethz.ch/content/dam/ethz/special-interest/baug/igp/photogrammetry-remote-sensing-dam/documents/pdf/learning-tracking-siamese.pdf https://ethz.ch/content/dam/ethz/special-interest/baug/igp/p... 2. https://zpascal.net/cvpr2018/He_A_Twofold_Siamese_CVPR_2018_paper.pdf https://zpascal.net/cvpr2018/He_A_Twofold_Siamese_CVPR_2018_... 3. http://openaccess.thecvf.com/content_ECCV_2018/papers/Yunhua_Zhang_Structured_Siamese_Network_ECCV_2018_paper.pdf http://openaccess.thecvf.com/content_ECCV_2018/papers/Yunhua...
kevinsundar 7y ago
I've found that IOU based tracking is surprisingly reliable (my dataset was primarily surveillance footage with a static camera) if you take measures to smooth the data and predict object motion using a kalman filter.
- adipandas 7y ago
  Kalman filter is also one good way to reliably keep track of objects under significant obstruction.

nullbyte 7y ago

PyImageSearch is a fantastic resource for learning this stuff. Highly recommend it.

mbpqd 7y ago

Question for HN experts: what would be the best approach for object-detection when the object I am trying to detect is a specific image (not a class of objects) printed on a 2D surface (therefore it could be partially folded / curved, but still more of a distorted 2D object than a complex 3D one)?

mendeza 7y ago
I would check out template matching or using OpenCV feature extractors such as SIFT, SURF, ORB Template Matching scans an image with your template, smaller image and finds the location with highest cross correlation: https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_imgproc/py_template_matching/py_template_matching.html https://opencv-python-tutroals.readthedocs.io/en/latest/py_t... Template matching assumes the example image wont deform or take on different orientations. Popular Feature Extraction finds and derive more robust features that help find objects in the scenarios of distortion, orientation, and scale. Link for feature detection and matching: https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_feature2d/py_matcher/py_matcher.html https://opencv-python-tutroals.readthedocs.io/en/latest/py_t... Pyimagesearch does a cool method where they enable template matching to work at different scales. It ignores color information (by using edge detection), which could be a critical feature if the object has specific color: https://www.pyimagesearch.com/2015/01/26/multi-scale-template-matching-using-python-opencv/ https://www.pyimagesearch.com/2015/01/26/multi-scale-templat...

alexcnwy 7y ago

I've built several object detection systems and you definitely want to fine-tune your model to get good results, even if the object class you want to track is available in a pre-trained model (often it isn't). For example, pre-trained models don't work too well at detecting people in surveillance footage when the camera is mounted on the ceiling and people are at an angle in the footage. Off-the-shelf object detection models also don't work well in poor lighting conditions but can work well if you fine-tune one of the pre-trained models for your dataset and lighting conditions. Here's a great and fun tutorial on how to fine-tune an object detection model using TensorFlow (it's a surprisingly fiddly process but worth the headache): https://towardsdatascience.com/how-to-train-your-own-object-detector-with-tensorflows-object-detector-api-bec72ecfe1d9 https://towardsdatascience.com/how-to-train-your-own-object-... I suspect you can use your fine-tuned model with this repo by replacing the `get_ssd_model.sh` compiled graph with your fine-tuned model graph. It's also a fun exercise to implement the tracking yourself - here's a solid tutorial to get you started: https://www.pyimagesearch.com/2018/08/13/opencv-people-counter/ https://www.pyimagesearch.com/2018/08/13/opencv-people-count...

adipandas 7y ago
The top-down view problem is what I was facing when I was trying to use a detection model for a project with quadcopters. For this I had to train my object detector with the data we collected from that view. YOLO was used for this work. Yes, replacing the fine-tuned model with compiled graph should work. I tried to keep it simple and intuitive to change the model so as to allow the use of any other custom models.