3 ms·

I'm the creator here. Yeah, 20 K80s is a bit excessive. That's because cortex (cortexlabs), which is the ML-model-deployment platform didn't initially have mult

I'm the creator here. Yeah, 20 K80s is a bit excessive. That's because cortex (cortexlabs), which is the ML-model-deployment platform didn't initially have multiprocessing on each of their replicas - so I was bound to using just one CPU per GPU. AWS has instances with 4, 8, 16 vCPUs and so on. Once cortex started supporting gunicorn (still yet unreleased but present on their master branch), I was able to reduce the number of GPUs significantly. Finally, the detector only needs 2 T4 GPUs and the identification part is the most expensive one (10 T4s GPUs) for a grand total of 12. Converting the models to use single-precision could further reduce the need to about 1.5 GPUs for T4s or just 1.2 GPUs for V100 - which in both cases it would still mean using 2 GPUs.

dheera 7y ago
Consider using a smaller, lighter-weight network (e.g. TinyYOLO) and object tracking (instead of running inferences on every camera frame) for faster throughput -- I imagine you should be able to get through with <1/4 of a V100 and still real-time for all practical cases. You can also customize the network to your use case, e.g. you don't need YOLO's default 5 anchor box sizes if you know the thing you're detecting is a license plate. Also, profile your code and see where your bottleneck is. If your bottleneck is at NMS for example there are things you can do to speed it up. I've seen a lot of cases where the neural network runs fast but there's a lot of Python bloat for pre/post-processing -- not sure about yours without seeing code. You really should be able to run a license plate detector/reader on something a lot smaller than a V100. A Xavier or quite possibly even a Jetson Nano would very likely be good enough if you use it well.
- xFlynnRider 7y ago
  I like all of your suggestions. I've been thinking about using TinyYOLOv3 as well. Provided the training set is considerably bigger than my own (I've created about ~550 samples and fine-tuned the model with them), you could end up with a very capable detection system that uses very few resources. Object tracking is yet a very good idea. I will consider it. Anchor-box tuning is another very good idea. Also, the CRAFT text detector that I'm using should IMHO be removed. Instead just use a very well trained text recognizer (like the CRNN I'm using). The text detector is expensive computationally since it's based on the VGG-16 model. Then convert the models to use mixed-precision. All in all, I think the performance improvements can be anywhere between 1 and 2 orders of magnitude.
SergeAx 7y ago
I believe you may achieve the same result locally using https://github.com/openalpr/openalpr https://github.com/openalpr/openalpr and cut your AWS and cell bills to exactly zero. It has Tesseract and OpenCV inside. Would love to see it as a part two of the article!
- xFlynnRider 7y ago
  I've come across that. And it looks awesome! Might give it a chance, why not. I see Tesseract's OCR engine is based on LSTM networks.

dheera 7y ago

Consider using a smaller, lighter-weight network (e.g. TinyYOLO) and object tracking (instead of running inferences on every camera frame) for faster throughput -- I imagine you should be able to get through with <1/4 of a V100 and still real-time for all practical cases. You can also customize the network to your use case, e.g. you don't need YOLO's default 5 anchor box sizes if you know the thing you're detecting is a license plate. Also, profile your code and see where your bottleneck is. If your bottleneck is at NMS for example there are things you can do to speed it up. I've seen a lot of cases where the neural network runs fast but there's a lot of Python bloat for pre/post-processing -- not sure about yours without seeing code. You really should be able to run a license plate detector/reader on something a lot smaller than a V100. A Xavier or quite possibly even a Jetson Nano would very likely be good enough if you use it well.

xFlynnRider 7y ago
I like all of your suggestions. I've been thinking about using TinyYOLOv3 as well. Provided the training set is considerably bigger than my own (I've created about ~550 samples and fine-tuned the model with them), you could end up with a very capable detection system that uses very few resources. Object tracking is yet a very good idea. I will consider it. Anchor-box tuning is another very good idea. Also, the CRAFT text detector that I'm using should IMHO be removed. Instead just use a very well trained text recognizer (like the CRNN I'm using). The text detector is expensive computationally since it's based on the VGG-16 model. Then convert the models to use mixed-precision. All in all, I think the performance improvements can be anywhere between 1 and 2 orders of magnitude.

SergeAx 7y ago

I believe you may achieve the same result locally using https://github.com/openalpr/openalpr https://github.com/openalpr/openalpr and cut your AWS and cell bills to exactly zero. It has Tesseract and OpenCV inside. Would love to see it as a part two of the article!

xFlynnRider 7y ago
I've come across that. And it looks awesome! Might give it a chance, why not. I see Tesseract's OCR engine is based on LSTM networks.