6 ms·

I am one of the people who helped analyze the results of the mentioned ILSVRC challenge. In particular, I performed an experiment comparing Google's performance

I am one of the people who helped analyze the results of the mentioned ILSVRC challenge. In particular, I performed an experiment comparing Google's performance to that of a human a week ago and wrote up the results in this blog post: http://karpathy.github.io/2014/09/02/what-i-learned-from-competing-against-a-convnet-on-imagenet/ http://karpathy.github.io/2014/09/02/what-i-learned-from-com... TLDR is that it's very exciting that the models are starting to perform on par with humans (on ILSVRC classification at least), and doing so on orders of milliseconds. The included page also has a link to our annotation interface where you can try to compete against their model yourself, and see its predictions and mistakes.

mrfusion 12y ago
Thanks for being here. Why do you think the results improved so much this year?
- karpathy 12y ago
  Google hasn't released details about their model yet, but the VGG team in close 2nd place had a very simple and beautiful "vanilla" CovnNet model, but with more careful hyperparameter settings and training/testing protocol. In other words, the source of improvement are tweaks on the original Krizhevsky architecture from 2012, not completely new and unexpected ideas. That is not to belittle the contribution - these experiments take forever to run and require very good practices and intuitions for what to try next. Karen released the details of the model only yesterday on arXiv (http://arxiv.org/pdf/1409.1556v1.pdf http://arxiv.org/pdf/1409.1556v1.pdf) More specifically, as can be seen in the paper, it seems that very deep stacks of conv/conv/pool modules with tiny 3x3 filters work well (which is satisfying because it's really simple and beautiful), and from being more thorough with the training and testing protocol (data augmentations, averaging, multiscale approaches in both train/test time, etc). Google will release details of their method on Sept 12 so we'll know more. From their abstract, it seems they have a more significant departure from a basic convnet architecture.
mentat 12y ago
So what the process for handling misclassification? http://cs.stanford.edu/people/karpathy/ilsvrc/val/ILSVRC2012_val_00042717.JPEG http://cs.stanford.edu/people/karpathy/ilsvrc/val/ILSVRC2012... is definitely a grand piano not an upright which is the "official" answer. Edit: http://cs.stanford.edu/people/karpathy/ilsvrc/val/ILSVRC2012_val_00042797.JPEG http://cs.stanford.edu/people/karpathy/ilsvrc/val/ILSVRC2012... is also misclassified. There isn't a bee house there. Closest thing is a barbecue. Edit 2: http://cs.stanford.edu/people/karpathy/ilsvrc/val/ILSVRC2012_val_00042836.JPEG http://cs.stanford.edu/people/karpathy/ilsvrc/val/ILSVRC2012... appears to be a building based on scaling clues.
- danbruc 12y ago
  http://en.wikipedia.org/wiki/The_Longaberger_Company http://en.wikipedia.org/wiki/The_Longaberger_Company
deleted 12y ago
[deleted]
possibilistic 12y ago
I hope I'm not too late to your thread to ask questions. I find your field absolutely fascinating, though I've never taken my interest further than basic undergrad image processing or simple machine learning. I wish I had more time to follow all of my interests. Anyway, this one particular gray area has been bugging me, and I've been hoping to run into a researcher or someone that has the appropriate context to clarify it for me. It's not technical question, per se.--it's a set of questions that build up to an uncertainty related to model training, reuse, and sharing. (I hope I'm asking the relevant questions...) I don't expect answers to all of this as it would take way too much of anyone's time, but maybe you can pick a few key points and discuss. I'm also really dying to get feedback on the last part as to its feasibility. 1. Do computer vision / modeling researchers typically share common training and evaluation data sets, or is it mostly kept proprietary? If good data sets do exist out in the open, do they undergo continual improvement, or are they frozen? Are these training sets typically amenable to only one type of training--ie., black box, offline vs online, etc.? Does it take a lot of practice/skill to know how much to hold back for post-training evaluation? Do you have an estimate for how much time and effort is involved in manually annotating and curating these data sets? 2. Once an algorithm is developed, does a model get trained under different parameters? Does tweaking those parameters lead to vastly different results? Are there typically distinct optima for a given classification task, or can it vary? Does the training set have a big impact on the performance of the algorithm? Which is more important, the training data or the algorithm? 3. Once models undergo offline training, are they fast to run? Is there a typical runtime complexity, or do different types of models operate significantly differently under the hood (by principle or by computational complexity)? Can you run an extremely complicated and robust model with a ton of classification outputs in sub-millisecond time on commodity hardware? 4. Can trained models be packaged and redistributed as a kind of "open source"? Are there any obvious barriers preventing this, such as the existence (or proliferation) of patents in your field? Do computer vision researchers like to share their code / results? If it's not a common practice to share code, would a large number of researchers be downright opposed to having their (patentable) algorithms and (copyrightable?) models made available to others? 5. Are trained models too complicated for the layman to use and produce good, consistent results? (For our purposes, I would consider a layman to be someone with at least some understanding of basic computer science, including exposure to data structures, algorithms, and a little bit of mathematical ability. Perhaps none of this too deep.) 6. Highly related to the last question, would there be a lot of specialized knowledge required to tweak model input parameters? Would these parameters correlate to mathematical operations? Would there be any arcane and seemingly arbitrary weights to adjust that are deeply encoded into the model itself? (Not sure if I'm far left field here or not.) I think what I'm ultimately hitting at is that it would be freaking awesome if there were a good set of robust pre-trained classifier models available for the layman programmer. Models that continually undergo development as improvements are made to training sets, algorithms, the literature--what have you. Please tell me if this kind of thing already exists. Anyway, I feel like I'm about to go on a long-winded talk about a technology I've been imagining. It's something along the lines of a specification that allows for the broad spectrum sharing of reusable, generic, containerized ML and classifier training results with others. In the world I envision, you would share trained models and import them just as you would code libraries. Let me reiterate: if this sort of thing happens to exist in the wild already, please point me to it! I can think of tons of uses. :) Note: replied to my own comment because original text was too long.

mrfusion 12y ago

Thanks for being here. Why do you think the results improved so much this year?

karpathy 12y ago
Google hasn't released details about their model yet, but the VGG team in close 2nd place had a very simple and beautiful "vanilla" CovnNet model, but with more careful hyperparameter settings and training/testing protocol. In other words, the source of improvement are tweaks on the original Krizhevsky architecture from 2012, not completely new and unexpected ideas. That is not to belittle the contribution - these experiments take forever to run and require very good practices and intuitions for what to try next. Karen released the details of the model only yesterday on arXiv (http://arxiv.org/pdf/1409.1556v1.pdf http://arxiv.org/pdf/1409.1556v1.pdf) More specifically, as can be seen in the paper, it seems that very deep stacks of conv/conv/pool modules with tiny 3x3 filters work well (which is satisfying because it's really simple and beautiful), and from being more thorough with the training and testing protocol (data augmentations, averaging, multiscale approaches in both train/test time, etc). Google will release details of their method on Sept 12 so we'll know more. From their abstract, it seems they have a more significant departure from a basic convnet architecture.

mentat 12y ago

So what the process for handling misclassification? http://cs.stanford.edu/people/karpathy/ilsvrc/val/ILSVRC2012_val_00042717.JPEG http://cs.stanford.edu/people/karpathy/ilsvrc/val/ILSVRC2012... is definitely a grand piano not an upright which is the "official" answer. Edit: http://cs.stanford.edu/people/karpathy/ilsvrc/val/ILSVRC2012_val_00042797.JPEG http://cs.stanford.edu/people/karpathy/ilsvrc/val/ILSVRC2012... is also misclassified. There isn't a bee house there. Closest thing is a barbecue. Edit 2: http://cs.stanford.edu/people/karpathy/ilsvrc/val/ILSVRC2012_val_00042836.JPEG http://cs.stanford.edu/people/karpathy/ilsvrc/val/ILSVRC2012... appears to be a building based on scaling clues.

danbruc 12y ago
http://en.wikipedia.org/wiki/The_Longaberger_Company http://en.wikipedia.org/wiki/The_Longaberger_Company

deleted 12y ago

[deleted]

possibilistic 12y ago

I hope I'm not too late to your thread to ask questions. I find your field absolutely fascinating, though I've never taken my interest further than basic undergrad image processing or simple machine learning. I wish I had more time to follow all of my interests. Anyway, this one particular gray area has been bugging me, and I've been hoping to run into a researcher or someone that has the appropriate context to clarify it for me. It's not technical question, per se.--it's a set of questions that build up to an uncertainty related to model training, reuse, and sharing. (I hope I'm asking the relevant questions...) I don't expect answers to all of this as it would take way too much of anyone's time, but maybe you can pick a few key points and discuss. I'm also really dying to get feedback on the last part as to its feasibility. 1. Do computer vision / modeling researchers typically share common training and evaluation data sets, or is it mostly kept proprietary? If good data sets do exist out in the open, do they undergo continual improvement, or are they frozen? Are these training sets typically amenable to only one type of training--ie., black box, offline vs online, etc.? Does it take a lot of practice/skill to know how much to hold back for post-training evaluation? Do you have an estimate for how much time and effort is involved in manually annotating and curating these data sets? 2. Once an algorithm is developed, does a model get trained under different parameters? Does tweaking those parameters lead to vastly different results? Are there typically distinct optima for a given classification task, or can it vary? Does the training set have a big impact on the performance of the algorithm? Which is more important, the training data or the algorithm? 3. Once models undergo offline training, are they fast to run? Is there a typical runtime complexity, or do different types of models operate significantly differently under the hood (by principle or by computational complexity)? Can you run an extremely complicated and robust model with a ton of classification outputs in sub-millisecond time on commodity hardware? 4. Can trained models be packaged and redistributed as a kind of "open source"? Are there any obvious barriers preventing this, such as the existence (or proliferation) of patents in your field? Do computer vision researchers like to share their code / results? If it's not a common practice to share code, would a large number of researchers be downright opposed to having their (patentable) algorithms and (copyrightable?) models made available to others? 5. Are trained models too complicated for the layman to use and produce good, consistent results? (For our purposes, I would consider a layman to be someone with at least some understanding of basic computer science, including exposure to data structures, algorithms, and a little bit of mathematical ability. Perhaps none of this too deep.) 6. Highly related to the last question, would there be a lot of specialized knowledge required to tweak model input parameters? Would these parameters correlate to mathematical operations? Would there be any arcane and seemingly arbitrary weights to adjust that are deeply encoded into the model itself? (Not sure if I'm far left field here or not.) I think what I'm ultimately hitting at is that it would be freaking awesome if there were a good set of robust pre-trained classifier models available for the layman programmer. Models that continually undergo development as improvements are made to training sets, algorithms, the literature--what have you. Please tell me if this kind of thing already exists. Anyway, I feel like I'm about to go on a long-winded talk about a technology I've been imagining. It's something along the lines of a specification that allows for the broad spectrum sharing of reusable, generic, containerized ML and classifier training results with others. In the world I envision, you would share trained models and import them just as you would code libraries. Let me reiterate: if this sort of thing happens to exist in the wild already, please point me to it! I can think of tons of uses. :) Note: replied to my own comment because original text was too long.