7 ms·

Current state of the art in objects classification

90 points by ZeljkoS 8y ago

Dzugaru 8y ago
Many years of "human optimization" on the same old datasets must lead to severe overfitting, so these results should be taken with a grain of salt. Related research: https://arxiv.org/abs/1806.00451 https://arxiv.org/abs/1806.00451
- bonoboTP 8y ago
  Have you read the linked paper? It finds no evidence of such overfitting happening, despite common fears of this phenomenon. Quote: "This shows that the current research methodology of “attacking” a test set for an extended period of time is surprisingly resilient to overfitting."
yorwba 8y ago
> Last updated on 2016-02-22. Given the speed of development in machine learning, the numbers are probably outdated.
bitL 8y ago
Must be very old, DenseNet-BC is nowhere mentioned...
dmm 8y ago
I see object detection systems often compared by accuracy percentages but rarely by speed. I ask because I'm building a surveillance system that uses object detection to answer simple questions, like: "Is there a squirrel on my bird feeder?", "Is my car in the garage?", "Is there a package on my front step?". Right now I'm playing with Darknet and yolo from pjreddie because he emphasized performance and I'm getting about 69fps on my 1080ti. Is speed not considered important by researchers at this time? Is there any system that provides decent performance on CPUs? Dedicating $700 gpus to detect squirrels on the bird feeder is fine for crazy people like me but others may find that excessive.
- kajecounterhack 8y ago
  Inference speed and net size is definitely considered important and some researchers are definitely tackling it (e.g https://arxiv.org/abs/1602.07360 https://arxiv.org/abs/1602.07360). However it's more of an engineering area (meaning lots of engineers are working on it) and less of a research area (meaning fewer papers get produced per work) because there's not much in the way of performance-per-gflop benchmarks or whatever. CPU performance is good-ish when you're doing sequential stuff e.g it's not bad at old-school RNNs, but for image recognition using a cheap graphics unit is still going to be better, and if the question is "can I use cheaper hardware and get good performance" the answer is yes, you just have to do some tricks (e.g distillation). Plenty of image recognition projects on the cheap raspberry pi and its graphics unit.
- dekhn 8y ago
  Inference Speed is very important. The TensorFlow object detection model zoo lists timings for their models; we picked our model to meet timing demands on a cell phone. I don't think CPUs are going to give you decent performance unless you microoptimize the inference for the host architecture and even then the gains are marginal. If you want to avoid expensive GPUs, take a look at things like Intel's Neural Compute Stick, Google's Vision Kit and the Edge TPU. But you can also run models on modestly priced cell phones- we use $100 cell phones to do object detection and tracking, but it's not pretty.
- antpls 8y ago
  If you look for "tensorflow raspberry" on a search engine, you will find several pointers about running TF on cheaper hardware. You would probably get about 1 frame per second with it. You may also look at recent SnapDragon-powered dev boards, which have TF core accelerators, and provide a serious boost of FPS for less than $700. Also, using a cloud ML service is an option if you have only a few inferences/day to process
- bitL 8y ago
  BTW, try SSD, it was even a bit faster than YOLO2.
  - 8y ago
jamessb 8y ago
Similar data (but for a wider range of problems - not just object classification) was tabulated by the EFF's AI Progress Measurement project [1]; I made an alternative interface to visualise it [2]. [1]: https://www.eff.org/ai/metrics https://www.eff.org/ai/metrics [2]: https://jamesscottbrown.github.io/ai-progress-vis/index.html https://jamesscottbrown.github.io/ai-progress-vis/index.html
dontreact 8y ago
This is two years out of date. Current state of the art for CIFAR has about 1/4th of the errors https://arxiv.org/abs/1805.09501 https://arxiv.org/abs/1805.09501
spott 8y ago
Anyone know of something like this for face recognition?

90 points by ZeljkoS 8y ago

Many years of "human optimization" on the same old datasets must lead to severe overfitting, so these results should be taken with a grain of salt. Related research: https://arxiv.org/abs/1806.00451 https://arxiv.org/abs/1806.00451

bonoboTP 8y ago
Have you read the linked paper? It finds no evidence of such overfitting happening, despite common fears of this phenomenon. Quote: "This shows that the current research methodology of “attacking” a test set for an extended period of time is surprisingly resilient to overfitting."

yorwba 8y ago

> Last updated on 2016-02-22. Given the speed of development in machine learning, the numbers are probably outdated.

bitL 8y ago

Must be very old, DenseNet-BC is nowhere mentioned...

dmm 8y ago

I see object detection systems often compared by accuracy percentages but rarely by speed. I ask because I'm building a surveillance system that uses object detection to answer simple questions, like: "Is there a squirrel on my bird feeder?", "Is my car in the garage?", "Is there a package on my front step?". Right now I'm playing with Darknet and yolo from pjreddie because he emphasized performance and I'm getting about 69fps on my 1080ti. Is speed not considered important by researchers at this time? Is there any system that provides decent performance on CPUs? Dedicating $700 gpus to detect squirrels on the bird feeder is fine for crazy people like me but others may find that excessive.

kajecounterhack 8y ago
Inference speed and net size is definitely considered important and some researchers are definitely tackling it (e.g https://arxiv.org/abs/1602.07360 https://arxiv.org/abs/1602.07360). However it's more of an engineering area (meaning lots of engineers are working on it) and less of a research area (meaning fewer papers get produced per work) because there's not much in the way of performance-per-gflop benchmarks or whatever. CPU performance is good-ish when you're doing sequential stuff e.g it's not bad at old-school RNNs, but for image recognition using a cheap graphics unit is still going to be better, and if the question is "can I use cheaper hardware and get good performance" the answer is yes, you just have to do some tricks (e.g distillation). Plenty of image recognition projects on the cheap raspberry pi and its graphics unit.
dekhn 8y ago
Inference Speed is very important. The TensorFlow object detection model zoo lists timings for their models; we picked our model to meet timing demands on a cell phone. I don't think CPUs are going to give you decent performance unless you microoptimize the inference for the host architecture and even then the gains are marginal. If you want to avoid expensive GPUs, take a look at things like Intel's Neural Compute Stick, Google's Vision Kit and the Edge TPU. But you can also run models on modestly priced cell phones- we use $100 cell phones to do object detection and tracking, but it's not pretty.
antpls 8y ago
If you look for "tensorflow raspberry" on a search engine, you will find several pointers about running TF on cheaper hardware. You would probably get about 1 frame per second with it. You may also look at recent SnapDragon-powered dev boards, which have TF core accelerators, and provide a serious boost of FPS for less than $700. Also, using a cloud ML service is an option if you have only a few inferences/day to process
bitL 8y ago
BTW, try SSD, it was even a bit faster than YOLO2.
- 8y ago

jamessb 8y ago

Similar data (but for a wider range of problems - not just object classification) was tabulated by the EFF's AI Progress Measurement project [1]; I made an alternative interface to visualise it [2]. [1]: https://www.eff.org/ai/metrics https://www.eff.org/ai/metrics [2]: https://jamesscottbrown.github.io/ai-progress-vis/index.html https://jamesscottbrown.github.io/ai-progress-vis/index.html

dontreact 8y ago

This is two years out of date. Current state of the art for CIFAR has about 1/4th of the errors https://arxiv.org/abs/1805.09501 https://arxiv.org/abs/1805.09501

spott 8y ago

Anyone know of something like this for face recognition?