9 ms·
"Baidu does not show up in this year’s rankings. The company made more submissions than were permitted and ultimately apologized and fired the team leader who d
by wyc 11y ago
"Baidu does not show up in this year’s rankings. The company made more submissions than were permitted and ultimately apologized and fired the team leader who directed juniors to make the unacceptable submissions."
- escap 11y agosee http://dswalter.github.io/blog/machine-learnings-first-cheating-scandal/ http://dswalter.github.io/blog/machine-learnings-first-cheat... for some background on this.
- sherjilozair 11y agoMight be worth noting that it is very likely that Google, Facebook and Baidu did not even participate in this ImageNet. It is widely believed in the vision community that this dataset is at the end of its run, i.e. any subsequent improvements in performance are not due to scientific breakthroughs but through hyperparameter optimization and ensembling.
- Estragon 11y agoSurely that's not true until someone achieves human-level performance with their entry.
- abrichr 11y agoHuman level performance was surpassed earlier this year: http://arxiv.org/abs/1502.01852 http://arxiv.org/abs/1502.01852
- natch 11y agoI won't be surprised when human level performance is surpassed, but isn't that paper based on work that has the advantage of knowing the full test dataset in advance? You can say "yes but they only trained on the training data" but that doesn't rule out tweaking across several experiments and measuring them against the known test data, then cherry picking the experiment that was best overfit to that data, right? I'm not saying there are shenanigans here, just wondering how you know there are not.
- argonaut 11y agoThe test labels are not known to them. The test labels are held by a different team that tells them their test error rate. Furthermore, multiple, separate teams have surpassed human performance. FWIW, "human performance" is not as meaningful as most people make it out to be, because this is such a narrow task.
- tim333 11y agoThe paper mentions it's unfair to humans to have to tell say a coucal from an indigo bunting (both blue coloured birds). I've never heard of either so I guess I would have got that wrong but it's not the fault of my visual system.
- p1esk 11y agoBut you only have to guess right once in 5 tries!
- natch 11y agoI replied to abrichr's link. Did you look at it at all? They used a dataset from 2012. Are you saying that the test labels from a 2012 dataset have been kept secret until now? Really?
- anirudhgarg 11y agoThe article and the title specifically mentions Google.
- krasin 11y agoGoogle participated with ReCeption / Inception v3 model, see the results: http://image-net.org/challenges/LSVRC/2015/results http://image-net.org/challenges/LSVRC/2015/results In classification task, MSRA and ReCeption show very similar performance: 0.03567 vs 0.03581 top(5) error. The gap is much more drastic on the localization task: 0.090178 vs 0.195792 The residual learning presented by Microsoft Research seems to be a breakthrough, based on the early evidence. But yes, ImageNet needs to be updated to stay relevant.
- nl 11y agoGoogle entered with ReCeption and finished just after the Microsoft team in the "Ordered by classification error" section (which is the usual "headline" number). Baidu was banned. I don't think Facebook has ever done ImageNet? But I agree that most of the interesting work will happen outside ImageNet now that human performance has been comprehensibly surpassed. The only exception is non-Deep Learning based systems, where some people remain convinced that alternative approaches can match DL systems and have other advantages.
- mac01021 11y agoI bet there are breakthroughs to be had if they would compete on training time and computing resources, rather than just accuracy.
- deleted 11y ago[deleted]