5 ms·
It's interesting that, in the image showing the classes easiest and hardest for the algorithm, all the easy ones are animals and all the difficult ones are huma
by pavlov 12y ago
It's interesting that, in the image showing the classes easiest and hardest for the algorithm, all the easy ones are animals and all the difficult ones are human-made artifacts.
Does nature tend to create forms which are easily detected by relatively simple neural networks? The evolutionary explanation could be that this allows animals with primitive neural systems to more easily distinguish other members of their species.
- Houshalter 12y agoMuch more training time was spent on animals (10% of the dataset is literally just pictures of dogs) and much more variation among man made objects.
- bane 12y ago> The evolutionary explanation could be that this allows animals with primitive neural systems to more easily distinguish other members of their species. It's probably the other way around. Neural systems optimize for natural objects.
- pavlov 12y agoBut isn't the neural network (like the algorithm described in the article) a mathematical concept? Are the computer-based neural networks so closely modeled after biological equivalents that they would inherit such an optimization? (As you can tell, I know nearly nothing about neural networks, but curious to learn more...)
- bane 12y agoI have no idea. But I'd wager that cause and effect in natural systems has more to do with visual systems adapting to shapes than shapes of entire animals adapting to other species' visual systems.
- deleted 12y ago[deleted]
- kastnerkyle 12y agoDigital circuits? I mean, it is just some matrix multiplies and a nonlinearity (then stack to the moon). No "circuitry" is really involved until you get into recurrent networks, and even then that is just feedback. Not quite sure what you mean here. There have been experiments trying to encode information the way the brain does, they just haven't worked very well (or at least not as well). There is equivalency with PCA (well ZCA, a modified form of PCA) in cat and monkey brains, and likely others as well. See Sejnowski and Bell in [1] Also, PCA is an affine transform, so there is no reason it couldn't be incorporated/learned by the net itself. In fact, I think most nets these days eschew PCA/ZCA when they have sufficient data support. To clarify for others, this type of neural network has nothing to do with the brain. A neural network is really a "universal function approximator", and I actually prefer to call it as such. Our goal is to learn the best possible mapping of input -> label, through whatever means necessary. It turns out that learning hierarchies of features helps from both a learning aspect and a computational point of view. But a sufficiently wide single layer could do the same thing in theory. [1] http://www.ncbi.nlm.nih.gov/pubmed/9425547 http://www.ncbi.nlm.nih.gov/pubmed/9425547
- michaelochurch 12y agoBut isn't the neural network (like the algorithm described in the article) a mathematical concept? It is. It's a family of models that (a) can be expressed compactly using linear algebra and (b) can represent a large class of mathematical functions. (In fact, the family of neural nets can learn all continuous functions.) "Learning" is typically some variety of gradient descent (on the "error surface" defined by the training set) in the space of parameters. Are the computer-based neural networks so closely modeled after biological equivalents that they would inherit such an optimization? In my opinion, no. Biological neural networks are, in fact, a lot more complicated. They have time behavior, changing network topologies, and chemical influences (neurotransmitters) that play a major role. There's still a lot we don't know about them. I disagree with GP's contention. While the biological neural network was an inspiration for this class of mathematical models, and convolutional behavior (in ANNs, a regularization, or mechanism for reducing the dimensionality of the parameter space, sacrificing training-set performance but often improving test-set performance) may be used in our visual cortex, but artificial neural nets are quite different and, mathematically, most varieties are quite simple.
- ma2rten 12y agoI think it's because these human made object have no single form. For instance "letteropener" describes the function of the object, not the form. Compare that to "red fox" which is always gonna look more or less the same.
- kastnerkyle 12y agoYes, but the network is still differentiating between select breeds which means it is learning traits of the animal which are unique to the breed. And training at a higher level is perfectly doable, say "fox" instead of the fox breed. Ultimately, if you are able to look at an image and say "letter opener" there are features which differentiate this from a knife/can opener/whatever - these are the exact things a convolutional neural network should be able to use (in theory) and has nothing to do with the label, which is typically unimportant as long as it is unique and accurate. We could flip all the labels around and still get unique answers - the network is just learning a mapping from input -> some integer, and I would argue the variance in dog breeds and lighting in natural scenes is much trickier than the angle/shape of a letter opener. I still think it comes down to the composition of this particular dataset. Augmenting this with images scraped from online stores would be very interesting as it is fairly trivial to get huge numbers of images for anything that is typically sold online - I think Google is way ahead on this one!
- bunderbunder 12y agoIt's impossible to tell from the examples given in the article, but I wouldn't be surprised if the same classifier that gets 100% on "Blenheim Spaniel" and "Flat-coated Retriever" gets less than 100% on "Dog". It's a question of how visually coherent the category you're trying to learn is. From a purely visual perspective, the first two categories are relatively tightly bunched in the state space, whereas "dog" covers a diffuse cloud of appearances whose total range might even encompass the area where many non-dog animals also lie. Humans may rely on some additional semantic knowledge about different kinds of animal to produce an accurate classification. It's not entirely unlike how determining the meaning of the words in the phrase "eats shoots and leaves" can't be done reliably without incorporating contextual clues such as whether we were just talking about pandas or a murder in a restaurant. There may also be issues around how distinct the categories are from each other. A couple years ago yours truly picked up a letter opener off the table and used it to spread butter on his toast, much to the amusement of his hosts.
- theoh 12y agoI'd say it's an entropy thing. The principle of "correlation of parts" means images of animals have low entropy relative to, say, machines. Intuitively a machine can contain an almost arbitrary set of shapes and components, whereas animal forms are more constrained.
- kastnerkyle 12y agoSee my earlier answer - I think that it is all about the data. And most things humans design have a similar "correlation of parts" due to the human preference for symmetry. Not all the images are face on headshots - lots of running/action, off centered, etc. ImageNet is a very "real" dataset in that sense. Entropy isn't the right term here - low entropy would imply something about the compressibility of each image based only on whether it was a machine or a natural object, which I don't think is case.
- theoh 12y agoWell, if the effect is just experimental error, I agree that no explanation is needed. I was using entropy in the information theoretic sense, as you might assign a measure of entropy in bits to each character in a language. If part of an animal is "less surprising" I'd say it contributed less to the entropy of the whole thing. Maybe that's too woolly. There is a difference between complex mechanical and organic objects, though. See the universal machine tools illustrated on this page: http://www.lathes.co.uk/adcock%26shipleycombination/ http://www.lathes.co.uk/adcock%26shipleycombination/ There is no way you could identify the whole machine based on seeing a small part if an identical part recurs in many different machines. It's not holographic in the way animals are.
- deleted 12y ago[deleted]
- possibilistic 12y ago> Does nature tend to create forms which > are easily detected by relatively simple > neural networks? Poison markings and mimicry, for sure. Furthermore, large swaths of plant life in various spectra. reply
- kastnerkyle 12y agoChicken, egg. Maybe these things are easy because they are crucial to survival (and everything that was bad at recognizing these signs died), not due to being easier for simple visual processes.
- bunderbunder 12y agoBoth chicken and egg at the same time. Evolution's going to favor the solution that involves the least costly adaptations. That probably means simpler markings will be favored because it's less costly for the prey species to evolve them and it's less costly for the predator species to evolve an instinct to avoid them.
- kastnerkyle 12y agoIt could also be that there are more labeled examples of animal, or that the translation/rotation of an animal is less disruptive than than the same transform to a car or other object. I know for sure that ImageNet has a huge amount of animals in it, down to very select sub-breeds, while the object categories are usually at higher levels.