6 ms·
Some background: This competition (the "Large Scale Visual Recognition Challenge" aka "ImageNet") isn't just some random competition. This is the competition t
by nl 11y ago
Some background:
This competition (the "Large Scale Visual Recognition Challenge" aka "ImageNet") isn't just some random competition. This is the competition that gave rise to the recent explosion in interest in Neural Networks.
In recent years Google, Microsoft and Baidu have been one-upping each other to the point now where they are getting very close to better-than-human performance (ie, humans disagree with other humans more often than their systems disagree with the average human rating)[0].
Andrew Ng went to Baidu to start their team. I don't believe he is still involved in this challenge.
Baidu has been getting close and close to Google's performance. Every year so far Google has topped it at the right time, but Baidu has later passed the Google benchmark that year.
There were reasons to think that this could be the year they finally beat them. [1] is a story from January about the system that Baidu had built then.
Now this.
[0] http://karpathy.github.io/2014/09/02/what-i-learned-from-competing-against-a-convnet-on-imagenet/ http://karpathy.github.io/2014/09/02/what-i-learned-from-com...
[1] https://gigaom.com/2015/01/14/baidu-has-built-a-supercomputer-for-deep-learning/ https://gigaom.com/2015/01/14/baidu-has-built-a-supercompute...
- deleted 11y ago[deleted]
- the8472 11y ago> ie, humans disagree with other humans more often than their systems disagree with the average human rating[0]. I wonder why google has that new image categorization captcha when it's a task where computers fare better than humans.
- heartbreak 11y agoHow do you suppose they obtain the data to know how humans fare?
- deleted 11y ago[deleted]
- the8472 11y agoSure, it's great for training/testing their AIs. But if the result is AIs better than humans at the task then it doesn't make a great captcha anymore.
- mkaziz 11y agoHence the fancy "click if not a robot" captcha. :)
- GuiA 11y agoMaybe they actually don't care about the captcha aspect - they just found a good mechanism to easily acquire data from humans that no one will complain about because it's so entrenched.
- avz 11y agoIt still protects against spammers that cannot deploy such AI. Which is probably very close to all of them.
- nailer 11y ago> There were reasons to think that this could be the year they finally beat them. Then they shouldn't have cheated. This is the same company that didn't do anything to stop it's network being used to wreck GitHub for a week (and yes, it's possible to avoid the GFW injecting JS onto your site - buy space in a non-China DC).
- cddotdotslash 11y agoAssuming a Chinese company can take legal measures to bypass a Chinese attack implemented by the Chinese government.
- nailer 11y agoIt is not currently illegal for Chinese companies to buy space in a non-China DC.
- Retric 11y agoThey were just caught blatantly cheating. This brings quite reasonable questions about their past performance.
- jdmichal 11y ago> This brings quite reasonable questions about their past performance. I don't think this second claim is necessarily derivable from your first. Yes, they were caught cheating... By making multiple submissions of their own systems in order to faster derive which is the most promising. IMO, the performance they have achieved is still theirs to claim, because in the end they did create a system that reaches that performance.
- Retric 11y agohttp://en.m.wikipedia.org/wiki/Overfitting http://en.m.wikipedia.org/wiki/Overfitting Ideally, your final test data set should be completely separate from training data. Because even limited exposure quickly renders that test data meaningless. PS: In the end it's much like taking the same test with word for word identical questions a second time. Yes, it's a test of something, just not the intended subject matter.
- ninjin 11y agoIt is hard to emphasize how bad overfitting on the test set is. Your cheating analogy is accurate, and cheating in science is a serious matter. If it is unintentional, you should have your work rejected or retracted. If it is intentional, you should have your work rejected or retracted and, frankly speaking, you should most likely leave the field. Science is based on a huge deal of trust and violating that trust for your own short term gains is inexcusable. Even worse, any honest scientist will have find it more difficult to improve upon results that can be explained by overfitting. A good short description on several issues that plague the field is "Clever Methods of Overfitting". http://hunch.net/?p=22 http://hunch.net/?p=22
- gwern 11y agoIt does, but I think the past performance is probably legitimate: the hopscotch nature of the Imagenet gains (in which it seemed like every month a different deep-learning team would turn in a new record) means that we can be confident that the overall performance curve of all teams is genuine unless they are all colluding, and this particular method of cheating previously can be ruled out just by looking at the submission scatterplot and cumulative submissions in the official statement ( http://www.image-net.org/challenges/LSVRC/announcement-June-2-2015 http://www.image-net.org/challenges/LSVRC/announcement-June-... ) - we can be sure that competitors and previous records did not overfit like this because there just aren't a whole lot of previous submissions to the test set. So now the record is probably more like 4.5% than the 4% or whatever Baidu's final paper was claiming. To keep this in perspective, that's a few months or a year of progress, the Imagenet record is still human-equivalent, and deep learning is amazing.
- lars 11y ago> getting very close to better-than-human performance I think they've all surpassed the human-level baseline by now. There's some stuff in there, like telling different dog breeds apart, that humans are not very good at.
- lern_too_spel 11y agoAndrew Ng made several posts about this result on Google+, most of which have now been removed. The reshares are still available though (like https://plus.google.com/+HeikkiArponen/posts/TaTm9jpCxqx https://plus.google.com/+HeikkiArponen/posts/TaTm9jpCxqx). He doesn't appear to have worked on the project though -- the "Deep Image" paper just credits him for "many insightful conversations." Baidu's blog post on the result has also been removed. Their ImageNet team was led by Ren Wu. http://webcache.googleusercontent.com/search?q=cache:Fs_26P8RVQUJ:usa.baidu.com/baidu-researchs-deep-image-system-attains-best-imagenet-classification-result/+&cd=4&hl=en&ct=clnk&gl=us&client=ms-android-att-us http://webcache.googleusercontent.com/search?q=cache:Fs_26P8...