6 ms·

I think about this stuff a lot as part of my day job. I'll discuss the good and the bad. I would say the most important part of this paper is the 1. the Size an

I think about this stuff a lot as part of my day job. I'll discuss the good and the bad. I would say the most important part of this paper is the 1. the Size and 2. the Number of images. The size of the model isn't huge. Most people don't realize that computer vision research is done using tiny sets of tiny images. Think hundreds to thousands of images that are 32x32 grayscale or black and white. Google did 10 million 200x200 color images. For a conservative comparison human vision can be thought of as two 15 megapixel cameras capturing at 30 frames per second. (The real numbers are higher and much more complicated.) I suspect that given this amount and quality of input data most machine learning algorithms would have produced better than state of the art results. Now the bad. Their model totally ignored time. Most computer vision researchers work with static images. In biology there is no such thing as a static image. Everything moves. You move, your eyes move, the environment moves. The brain, as an exception, can understand static images, but the rule is motion. A majority of features in biological vision are tied to motion. Light/dark change cells, moving edge change cells. A huge portion of the information you derive from sight comes from the temporal context of a given moment. The change from moment A to moment B is directly encoded and learned and is often more important than the state at A or the state at B. I'm not faulting them, in fact, I am extremely happy with their results, but I want to point out that these results can and will get much better in the near future as the input size increases and people start to integrate temporal features into their models.

fghh45sdfhr3 14y ago
On the one hand I am an experienced software engineer with a computer science degree. On the other hand I was never interested in AI and I basically know "nothing" about AI, computer vision and artificial neural networks. AI always seemed too far off and bothered my engineering spidey sense for intractable problems. So I may embarrass myself with the following... The more software engineering experience I get the more neural networks bother me. I have not studied them in detail... and yet, they seem like a crappy solution. Throw a lot resources at something, add a bit of math and push a pile of data through. Does that ever lead to result that can't be bested though simpler more traditional methods? For example, the math behind how the network is organized and learns, that same math can be used to write algorithms which reveal patterns in data. Patterns which can then be studied further. Then there is the doing of things. What ever pattern is responsible for doing something in a network, the same function can be described much more simply by an algorithm. It just seems like neural networks would be interesting only if an alien spaceship with stupefying computing resources fell on us. And it fell on us in such a way that we could fully utilize the power of this new super computer while at the same time we could not learn anything new about algorithms or anything else. Only then does it seem like "train the neural network" is a smart strategy. In every other scenario is just seems like a decent if somewhat lazy start to an investigation. Really just a way to kick the tires of a problem. So did I just completely embarrass myself with my lack understanding of ANNs?
- kevin1024 14y ago
  Check out this (awesome) talk: http://www.youtube.com/watch?v=ZmNOAtZIgIk http://www.youtube.com/watch?v=ZmNOAtZIgIk There are a number of areas in which unsupervised feature learning outperforms the hand-programmed models we have been fine-tuning for decades.
- entropy_ 14y ago
  I don't understand why this got down-voted. Having read up about AI and looked into things like neural networks I can definitely tell that he's wrong. But surely the questions he asks are pertinent to the discussion, aren't they? For someone foreign to the field they actually make a lot of sense and I would be very hard pressed to tell him why he's wrong given that I only have very little knowledge myself. Surely whoever thought this post was so wrong as to merit down-voting could explain why that is and answer the man's questions?
  - adgar 14y ago
    People are dicks on the Internet.
- lmkg 14y ago
  If you try to design an algorithm, what you get is a model of how you think your system behaves. If you train a neural network, what you get is a model of how your system actually behaves. In other words, a neural network does not embed any assumptions or preconceived notions of how the system ought to work. Intuitively, not incorporating knowledge of how the worlds works ought to be a disadvantage, but in practice it turns out to be not just an advantage, but an embarrassingly large one. Neural Networks aren't the only common approach in modern AI research, but the other techniques (e.g. Support Vector Machines) have similar properties: they're dumb methods of finding correlations in data. The field of AI is slowly coming to grips with the fact that more training data trumps better algorithms. That said, I sympathize with your frustrations. You can solve a lot of problems by throwing more cycles at it, but you don't learn anything, either about the algorithms or about your target domain. For example, a human plays the game of Go differently than a machine. A human has an intuition of what moves are good and what are bad, and only considers a handful; a machine relies less on evaluating board position and more on brute-forcing through possible board configurations. As machines get more powerful, eventually the machines catch up to humans in ability, with that same approach. While their methods will become impressive in their efficacy, we learn nothing about how humans are so much better at evaluating board position and selecting moves to speculate on.
marshallp 14y ago
Your wrong about most conventional algorithms being able to get better results just by increasing image size. First, theyre isn't a dataset suitable for only supervised learning (the final supervised stage they - imagenet - only has a relatively small amount of images of each category). Second, higher resolution can make things worse in conventional supervised learning, you have to get more and more images to keep up with higher resolution (extra things in the image start confusing the algorithm, and even humans - more identifiable things in the photo mean you have to think more about what is the most important thing in the image). The unsupervised algorithm gets around the both problems. There existed unsupervised/clustering algorithms before, but none that worked well until geoff hinton's breakthrough in 2005 (for which he deserves a turing award).
- iandanforth 14y ago
  Thanks for the correction. I should not have said most. I tend to focus on ones that do have this property and forget about the vast number of supervised learning techniques.

fghh45sdfhr3 14y ago

On the one hand I am an experienced software engineer with a computer science degree. On the other hand I was never interested in AI and I basically know "nothing" about AI, computer vision and artificial neural networks. AI always seemed too far off and bothered my engineering spidey sense for intractable problems. So I may embarrass myself with the following... The more software engineering experience I get the more neural networks bother me. I have not studied them in detail... and yet, they seem like a crappy solution. Throw a lot resources at something, add a bit of math and push a pile of data through. Does that ever lead to result that can't be bested though simpler more traditional methods? For example, the math behind how the network is organized and learns, that same math can be used to write algorithms which reveal patterns in data. Patterns which can then be studied further. Then there is the doing of things. What ever pattern is responsible for doing something in a network, the same function can be described much more simply by an algorithm. It just seems like neural networks would be interesting only if an alien spaceship with stupefying computing resources fell on us. And it fell on us in such a way that we could fully utilize the power of this new super computer while at the same time we could not learn anything new about algorithms or anything else. Only then does it seem like "train the neural network" is a smart strategy. In every other scenario is just seems like a decent if somewhat lazy start to an investigation. Really just a way to kick the tires of a problem. So did I just completely embarrass myself with my lack understanding of ANNs?

kevin1024 14y ago
Check out this (awesome) talk: http://www.youtube.com/watch?v=ZmNOAtZIgIk http://www.youtube.com/watch?v=ZmNOAtZIgIk There are a number of areas in which unsupervised feature learning outperforms the hand-programmed models we have been fine-tuning for decades.
entropy_ 14y ago
I don't understand why this got down-voted. Having read up about AI and looked into things like neural networks I can definitely tell that he's wrong. But surely the questions he asks are pertinent to the discussion, aren't they? For someone foreign to the field they actually make a lot of sense and I would be very hard pressed to tell him why he's wrong given that I only have very little knowledge myself. Surely whoever thought this post was so wrong as to merit down-voting could explain why that is and answer the man's questions?
- adgar 14y ago
  People are dicks on the Internet.
lmkg 14y ago
If you try to design an algorithm, what you get is a model of how you think your system behaves. If you train a neural network, what you get is a model of how your system actually behaves. In other words, a neural network does not embed any assumptions or preconceived notions of how the system ought to work. Intuitively, not incorporating knowledge of how the worlds works ought to be a disadvantage, but in practice it turns out to be not just an advantage, but an embarrassingly large one. Neural Networks aren't the only common approach in modern AI research, but the other techniques (e.g. Support Vector Machines) have similar properties: they're dumb methods of finding correlations in data. The field of AI is slowly coming to grips with the fact that more training data trumps better algorithms. That said, I sympathize with your frustrations. You can solve a lot of problems by throwing more cycles at it, but you don't learn anything, either about the algorithms or about your target domain. For example, a human plays the game of Go differently than a machine. A human has an intuition of what moves are good and what are bad, and only considers a handful; a machine relies less on evaluating board position and more on brute-forcing through possible board configurations. As machines get more powerful, eventually the machines catch up to humans in ability, with that same approach. While their methods will become impressive in their efficacy, we learn nothing about how humans are so much better at evaluating board position and selecting moves to speculate on.

marshallp 14y ago

Your wrong about most conventional algorithms being able to get better results just by increasing image size. First, theyre isn't a dataset suitable for only supervised learning (the final supervised stage they - imagenet - only has a relatively small amount of images of each category). Second, higher resolution can make things worse in conventional supervised learning, you have to get more and more images to keep up with higher resolution (extra things in the image start confusing the algorithm, and even humans - more identifiable things in the photo mean you have to think more about what is the most important thing in the image). The unsupervised algorithm gets around the both problems. There existed unsupervised/clustering algorithms before, but none that worked well until geoff hinton's breakthrough in 2005 (for which he deserves a turing award).

iandanforth 14y ago
Thanks for the correction. I should not have said most. I tend to focus on ones that do have this property and forget about the vast number of supervised learning techniques.