8 ms·
The ImageNet dataset transformed AI research
- Maven911 9y agoImageNet is the big one on image recognition, is there other AI/ML/DL competitions that people have heard of ?
- matt4077 9y agoKaggle, The Netflix Challenge, CASP
- jonbaer 9y agoThe Loebner (on the NLP side) this year should be interesting - http://www.aisb.org.uk/events/loebner-prize http://www.aisb.org.uk/events/loebner-prize - on the AI planning side usually the StarCraft competitions - https://sscaitournament.com https://sscaitournament.com - interested to see how DeepMind performs.
- bllguo 9y agoslightly different, but wanted to add numer.ai to the great list others have commented
- marcelsalathe 9y agoThe non-profit platform https://www.crowdAI.org https://www.crowdAI.org currently hosts a NIPS 2017 challenge as well as a genetic challenge, and constantly adds new challenges. Btw, it's completely open source, and keen on having contributors to help build the open challenge platform the community wants the most.
- deepnotderp 9y agoMS COCO, although i think that's part of the imagenet contest now.
- yonilevy 9y ago(Disclaimer: I work at Nexar) Nexar just released Nexet, a large dataset of 55K road images (5K of which are labeled). I think it's an interesting one as the images were captured by mobile phones (setup as dashcams), so you get "real world" data. Of-course there's an accompanying competition coming up: https://www.getnexar.com/challenge-2/ https://www.getnexar.com/challenge-2/
- colmvp 9y agoThere's WIDERFace(http://mmlab.ie.cuhk.edu.hk/projects/WIDERFace/index.html http://mmlab.ie.cuhk.edu.hk/projects/WIDERFace/index.html) which is specific to facial recognition.
- PeterisP 9y agoOn NLP side, many subfields tend to run annual shared task competitions, e.g. SemEval (http://alt.qcri.org/semeval2017/index.php?id=tasks http://alt.qcri.org/semeval2017/index.php?id=tasks) or TAC (https://tac.nist.gov/ https://tac.nist.gov/) for semantic analysis, WMT (http://www.statmt.org/wmt17/index.html http://www.statmt.org/wmt17/index.html) for machine translation, etc etc.
- bra-ket 9y agoFYI Kaggle is hosting all 3 Imagenet challenges now: https://www.kaggle.com/c/imagenet-object-detection-from-video-challenge https://www.kaggle.com/c/imagenet-object-detection-from-vide...
- nightcracker 9y agoIs it possible to hold an image classification challenge without secret data that doesn't lead to overfitting?
- Govindae 9y agoYeah, just specify the test set and ask everyone not to train on it.
- komali2 9y ago>Matthew Zeiler built Clarifai based off his 2013 ImageNet win, and is now backed by $40 million in VC funding. I used Clarifai at a hackathon a couple months ago. It was certainly impressive, but I didn't really see the difference between it and Google Cloud Services AI whatever offerings, other than Google seemed to have a better handle on deployment and production scaling. I guess I'm wondering how companies like Clarifai and the couple other similar "AI API cloud service" companies intend to take on behemoths like Google?
- eanzenberg 9y agoNothing, as far as I can tell, clarifai isn't doing anything novel or pushing research into the field. It looks like an ego-project for their CEO.
- m_ke 9y agoThey launched almost two years before tensorflow and google vision APIs were released. At the time there were only a few labs doing deep learning research so an API seemed like the best way to make the technology accessible to more people. Publishing research papers is a great way to get acquired but it's not the best use of your time if you're a small startup that's trying to make money. They still do research but it makes it into the product instead of getting published.
- Mathnerd314 9y agoMaybe custom / bespoke model training, plus they can offer a lot better customer support than Google. A sucker is born every minute...
- eanzenberg 9y ago"Li said the project failed to win any of the federal grants she applied for, receiving comments on proposals that it was shameful Princeton would research this topic, and that the only strength of proposal was that Li was a woman." Jesus..
- scottLobster 9y agoStories like this are part of the reason I never went into academia. It's amazing how many highly educated people are judgmental and inflexible, and in academia one person like that can sink you. As an engineer in the private sector, worst case scenario I can find a better team/company to work with.
- ironic_ali 9y agoI picked up on that too. It's the biggest failing "high up" (their own arse) many academics have.
- contingencies 9y agoI found getting halfway decent funding since 1980 to be quite a chore. - Alan Kay (2016) ... from my fortune clone @ http://github.com/globalcitizen/taoup http://github.com/globalcitizen/taoup
- dmix 9y agoNot just a dataset but hacking community and contest: > Matthew Zeiler built Clarifai based off his 2013 ImageNet win, and is now backed by $40 million in VC funding. On a side note, this looks like a fun company to build: hacking on ML, exposing it via APIs, developers as customers, paid for on a resource basis like a VPS style per-use payment plan. It's interesting how many of these AI startups are raising huge rounds. The words AI seems to draw them in. At least this one has an obvious business model.
- eli_gottlieb 9y agoI was went to an AI talk at MIT CSAIL (the same Vicarious one I mentioned in another thread). Me and a guy got talking on the way out. He figured I seemed pretty clever, so he started asking me what sorts of companies to invest in. Turned out he was an investor with some VC firm. I did a little mention of the place I worked at the time (which isn't an AI company, but wanted to be when it was young), but didn't have a business card to give him at he time. I was actually kinda surprised an investor was attending a talk and saying, quite so uncritically, "hmmm, who do the smart people at talks say I should invest with?"
- relate 9y agoShameless plug: we have a workshop today @ CVPR 2017 that explores the classification task of ImageNet using crawled (and noisy) data: https://www.vision.ee.ethz.ch/webvision/workshop.html https://www.vision.ee.ethz.ch/webvision/workshop.html In contrast to ImageNet, only the validation and test data were manually labeled. Impressively, on such data, an accuracy of ~95% (top5) is achieved by the winning solution this year.
- businessdog 9y agoBuilt using the Stanford Dogs ImageNet dataset: https://dog.ceo/dog-api https://dog.ceo/dog-api
- jjawssd 9y agoNever forget about the most important person in all recent neural network object recognition research: Mark Everingham. Mark was the key member of the VOC project, and it would have been impossible without his selfless contributions. https://www.computer.org/csdl/trans/tp/2012/11/ttp2012112081.pdf https://www.computer.org/csdl/trans/tp/2012/11/ttp2012112081... http://www.bmva.org/w/doku.php?id=obituaries:mark_everingham http://www.bmva.org/w/doku.php?id=obituaries:mark_everingham Mark Everingham passed away in 2012, bringing the VOC project to an end.
- ouid 9y agoimagenet is a truly terrible dataset. Any image recognition task that humans are not >99.9% on is not an image recognition task.
- tanilama 9y agoA truly terrible dataset that leads to AlexNet/VGG/Neural Style Transfer/GoogleNet/ResNet, and facilitate research so much, is still a good dataset. Actually no, it is a phenomenal dataset. Not to mention, it is free and open to all. You can make assessment even if you are not an expert, but I would think you need more efforts to backup your claim a little bit to convince people and avoid downvoting.
- PeterisP 9y ago99.9% is an extremely high bar. People aren't machines, people make errors. If you asked them to tell whether a square is black or white, you would get less than 99.9%.
- ouid 9y ago99.9% is a very low bar for image recognition. if one out of every thousand times you saw a car you misclassified it as a chair, you would be dead.
- mturmon 9y agoIt sounds like you may not have experience in this domain? You're benchmarking an active, time-based task (a person "seeing a car" - implicitly allowing eye saccade, multiple images, context, depth) versus a static task ("classify this single image based on no context"). They are not comparable.
- ouid 9y agohave you actually tried the imagenet task yourself? The reason imagenet is popular is because it has allowed google to claim that they can beat human performance on image recognition by artifically lowering human performance on image recognition.
- contingencies 9y agoTangential anecdote: Incidentally the article mentions that ImageNet was conceived through WordNet[0]. I actually downloaded WordNet again last night with the idea to relocate qualifying adjectives from certain noun phrases in a system to a separate list of qualifiers, ie. "adj1 adj2 noun" -> {"noun",["adj1","adj2"]} What I found was that WordNet has crap coverage, eg. large numbers of real adjectives were unlisted. In my ancient Chinese translation work (hobby), I have basically concluded that Wiktionary[1] is the best thing going, and also easiest to contribute corrections and background data to. So I thought "why not adapt English Wiktionary data[2] to produce an English adjective list?". This proved far more complicated than a quick hack, since the only way the data seemed to be supplied was as a huge XML file mixing Wiki and XML markup.[3] In the end I downloaded a Wiktionary-derived dataset known as dbnary[4] and parsed wordlists out of that, resulting in a satisfying list of 103,125 English adjectives. Total time invested, under 2 hours. [0] http://wordnet.princeton.edu/ http://wordnet.princeton.edu/ [1] http://en.wiktionary.org/ http://en.wiktionary.org/ [2] https://dumps.wikimedia.org/backup-index.html https://dumps.wikimedia.org/backup-index.html [3] https://dumps.wikimedia.org/enwiktionary/20170720/enwiktionary-20170720-pages-articles.xml.bz2 https://dumps.wikimedia.org/enwiktionary/20170720/enwiktiona... [4] http://kaiko.getalp.org/about-dbnary/ http://kaiko.getalp.org/about-dbnary/
- PeterisP 9y agoBabelNet http://babelnet.org/ http://babelnet.org/ or Open Multilingual Wordnet (http://compling.hss.ntu.edu.sg/omw/ http://compling.hss.ntu.edu.sg/omw/) might be interesting for you. However, the prime use case of WN is as an verified inventory of machine readable senses, providing structured information about the semantic attributes of the word by providing the semantic links to other words/senses. Wictionary will tell you that a word is an adjective, but it doesn't tell a machine what does that adjective mean, how it relates to other adjectives. E.g., the wiktionary entry on 'corgi' has a few sentences that mention that it's a type of dog, but it's not structured enough for this information to be used, and if you just extract it yourself without manual verification then it won't have high accuracy; WN, on the other hand, has an explicit structured link (hypernym) between corgi and dog. (http://wordnetweb.princeton.edu/perl/webwn?o2=&o0=1&o8=1&o1=1&o7=&o5=&o9=&o6=&o3=&o4=&s=corgi&i=2&h=100#c http://wordnetweb.princeton.edu/perl/webwn?o2=&o0=1&o8=1&o1=...)
- unityByFreedom 9y agoLooking forward to when someone makes a significantly sized, fully labeled medical dataset. Currently, the main thing holding us back on diagnostics using machine learning isn't the software or hardware- it's the lack of labeled data, which is costly to prepare because it requires technologists to work alongside doctors. Still, the results could be huge. I think machine learning through medicine will save more lives than self driving cars.
- PeterisP 9y agoIt's highly problematic due to privacy issues. It would be a nightmare to get a public release for the underlying unlabeled data, especially since we know how well deanonymization can work.
- unityByFreedom 9y agoThere already is some that is publicly available, like the Kaggle lung cancer competition's dataset. I'm not saying it's easy on any front. It is possible, and it would be hugely beneficial to humans to have such a dataset. Raising awareness about the problem may be one step towards solving it.
- nonbel 9y agoThe problem with a lot of medical labeling is the labels are not very consistent/accurate to begin with. Just go to three doctors with the same symptoms and see what they say...
- unityByFreedom 9y agoThat is surmountable. You can use annotator agreement, and, algorithms don't need perfectly labeled data in order to improve. Any effort to create additional medical datasets would probably help enhance machine learning diagnostics. The data that you want to be perfect is the test set, which is smaller than training sets.