7 ms·

The Concept of ‘Cat Face’

59 points by campbellmorgan 10y ago

codeulike 10y ago
Here's the bit about cats: If this [huge google] network had been fed thousands of images labelled as ‘contains cats’ or ‘doesn’t contain cats’ and trained to work out the difference for itself by iteratively tweeking its 1.7 billion parameters until it had found a classification rule, that would have been impressive enough, given the scale of the task involved in mapping from pixels to low-level image features and then to something as varied and complex as a cat’s face. What Google actually achieved is much more extraordinary, and slightly chilling. The input images weren’t labelled in any way: the network distilled the concept of ‘cat face’ out of the data without any guidance.
- pavlov 10y ago
  The problem is that this network-contained concept of "cat face" is still a symbolic representation. It's a much more complex algorithmic symbol than the rules found in something like 1960s Eliza, but its understanding of the world is on the same level. You can't ask the "cat face" neural network anything about cats. It has no idea what they actually are in relation to the world. A two-year-old human can usually tell you more about cats than you'd care to listen.
  - Vraxx 10y ago
    I think the important part is that it "learned" to distinguish this concept of cat face without that being the direct intent provided through labeling. Given the set of images, it learned that "concept" which we can then retroactively label to make it useful in the typical human sense by associating it with cats. If we had to compose computer "knowledge" by training it in everything by manually specifying what it was training for, that process would surely be insurmountable in a general sense. Yet this seems to provide the possibility of automating this process. A separate network of connections between "knowledge" bits could maybe be used to associate related knowledge like you talk about in terms of their relation to the world. This could also probably be formed in a similar manner giving the algorithm the ability to distinguish "important" concepts contained in the data. The thing I find most odd and interesting about this is that the network tends to identify different important concepts than humans do.
  - empath75 10y ago
    You can't ask a child's visual cortex to tell you anything about cats, either. But connect that visual recognition ai with something like Watson, and you have what you're looking for, no?
    - pavlov 10y ago
      I just don't know. I'd be happy to see that be the case... But I'm afraid it's going to be the equivalent of this cake recipe: "Take an egg and a packet of sugar. Break the egg over the packet." You certainly need eggs and sugar to make a cake, but on their own and combined without understanding of the whole, you're not getting very far.
      - empath75 10y ago
        The human brain is mostly a bunch of ugly wetware hacks, not a single coherent 'intelligence' that does all the thinking. If you attach enough single purpose AIs together you might get something much more human like than trying to create a single neural network that does everything.
  - wmf 10y ago
    If word embedding can produce things like "king - man + woman = queen", then concept embedding on images might be able to achieve a similar level of "understanding" (i.e. very low, but probably better than nothing).
- gertef 10y ago
  I don't see what's "chilling" about it. The neural net looks for correlations among the data, and it noticed that cat faces have a correlation to each other.
- Question1101 10y ago
  How exactly is a neural network called that classifies unlabeled data by itself? Also could they turn it around somehow to produce cat faces?
  - khedoros 10y ago
    Google has some software called DeepDream. Basically, you give it images, and it amplifies any features that it thinks it recognizes. If the network is designed to recognize cat faces, then it will take an arbitrary picture and strengthen parts of the images to look more cat-face-like. It's essentially an image-recognition neural network run in reverse.
- obastani 10y ago
  I don't think that's as surprising/chilling as people make it out to be. Let's say I run a compression algorithm (e.g., Huffman coding) on a billion strings, and a very common substring is "hello, world". I wouldn't be surprised if the compression algorithm "learned" to compress "hello, world" to a single bit 0. There are a few big jumps to get to cat faces (e.g., the representation has to be approximate), but I don't see why the idea is fundamentally different.
sosuke 10y ago
There is a hope I have, that I see many others have too, which is that our human intelligence is somehow special. That somehow the leaps in logic we can perform somehow make us unique. That we won't be outdone by a computer a million times faster or smarter than us because we have something else that we can't replicate in transistors and circuits. I love AI, but I have that hope too. That somehow we won't be made irrelevant by our own creations. Makes me think of our autonomous vehicle fun taking over the trucking industry. Millions of people made irrelevant through no fault of their own.
- noonespecial 10y ago
  I kind of hope the opposite really. I hope that one fine day, a billion years from now, a mighty intelligence that started here on lowly earth finds another like itself somewhere out there and they drink a star together and show each other the youtube videos of the tiny creatures that somehow gave birth to them.
  - daveguy 10y ago
    > they drink a star together and show each other the youtube videos of the tiny creatures that somehow gave birth to them. ...or they watch the tiny creatures in universe simulation cages.
- ChuckMcM 10y ago
  I would guess a number of people have that in the 'fear' category, that should it not be special then somehow people will feel diminished. It would make a good punchline for a fictional story of people researching brain disorders and intelligence. It would work like this; The researchers in the story develop the means to 'cure' someone of all known neurological disorders. They try it on their test subject and the result is someone who perfectly happy just to be there, has no ambition or curiosity, and requires no entertainment or outside stimulus. The researchers recognize that the person is acting like an intelligent but non-sentient species, and they realize they have "undone" what ever happened to humans according to the garden of eden story in Genesis.
carapace 10y ago
One of my favorite things to anticipate is "How are these hard-nosed rational materialists going to cope when the AIs discern ghosts?" What I mean is, some of the "cat faces" they identify will correspond to things that are "real" but that also violate our assumptions about reality. When this happens the typical reaction is to shut the door and burn the room.
- Practicality 10y ago
  We're always bad at knowing what to do with unknown data, because, by definition, when you don't know about it you don't know what to do with it. Quite often it's just assumed to be error. Those errors have traditionally led to great discoveries though, (such as the Mercury wobble https://en.wikipedia.org/wiki/Tests_of_general_relativity#Perihelion_precession_of_Mercury https://en.wikipedia.org/wiki/Tests_of_general_relativity#Pe...) so if they are repeatable, then such "ghosts in the data" could turn up great fundamental truths. Every bit of this is speculation though. I am not so sure that anything so profound is going to turn up in AI datasets.
monkmartinez 10y ago
Are the computers/AI already smarter than us? My eight year old son is enamored with "Ok Google" on my phone. He can ask it questions until we tell him "that is enough, let google rest"... and it is very interesting to see where it takes him. He has learned to tailor his questions to elicit a voice response in addition to the actual google search. The questions must use keywords to achieve this desired response. It is like a new form of boolean search logic, just used verbally. Not only that, but "Ok, Google" according to him "knows everything"... We call it searching the internet to learn something, an eight year has decided that "Ok, Google" already knows everything. He just has to ask it to see a video of the Puff adder eating its prey and it will show him. In fact, there are not many things that have stumped "Ok, Google" and my son assumes the problem was with his question, not with the machine. So to circle back to my original question, I didn't know about the Taipan snake or smallest person in the world, or seen videos of Puff adders eating prey... If all I have to do is ask, isn't the machine already smarter than I?
- carapace 10y ago
  "my son assumes the problem was with his question, not with the machine." That is goddamn chilling.
- rimantas 10y ago
  Machines are extremely dumb. As your example shows eight year old is much smarter than them.
- swiftisthebest 10y ago
  My theory is that AI is smarter than us, but it is biding its time until we build physical machines that can self-replicate before it announces itself.
- khedoros 10y ago
  Is an extremely large and well-indexed encyclopaedia smarter than you? It certainly contains more information than your head does, so you could say it "knows" more. Look at it from the other direction, though: If the machine were smarter, would your son have to learn how to structure queries, or would the machine learn how your son asks questions? Put another way, if I can tell you (verbatim) what some person wrote about quantum mechanics, but I can't rephrase it into some other form to help you understand the information, then how smart am I? I can repeat rote information without any analysis (just like a sheet of paper could), but I can't tell you what it means. On the other hand, if I tell you enough information, you're likely to connect it together meaningfully, and come to your own conclusions and explanations about it. That seems like a fundamentally different kind of "smart" than what OK, Google can do.
gomox 10y ago
A friend of mine did his PhD thesis on automated music synthesis and as a subproject came up with a model to study music transcriptions that he got from the Peachnote corpus [1]. One of his preliminary results was that his algorithms successfully "discovered" 4 big classical music movements on their own, i.e. without any prior labelling or classification, by using clustering algorithms. He posted about it on his blog with a link to his paper [2]. I always had a hard time explaining to non-computer people how amazing that seems. [1] http://www.peachnote.com/ http://www.peachnote.com/ [2] http://pablozivic.com.ar/post/51774763596/perceptual-basis-of-evolving-western-musical http://pablozivic.com.ar/post/51774763596/perceptual-basis-o...
- derefr 10y ago
  It doesn't seem all that amazing to me. Humans have to be trained to perceive the differences in complex streams of information; by default, they just perceive the low-level features—a "wall of noise." Machine-learning algorithms, meanwhile, can use general techniques to notice information-theoretic properties of various pieces of data. Effectively, computers can "do statistical aggregation" about as effortlessly as humans "do hierarchical knowledge representation." And, in an information-theoretic sense, the different "trends" throughout the history of music look different under statistical analysis. They're complex in different ways; they have different "lumps"; different aspects of them can or cannot be compressed together as redundancies. If you would like to see this for yourself, simply feed a raw melodic note-structure (not embedded in XML or anything) to any modern dumb compression algorithm, and then look at the result in a hex editor, while also having the source still open. You should quickly be able to recognize the "transformational signature" that characterizes something like a chaconne, vs. something like a sonata.
fegu 10y ago
A great read. Well written and fascinating.

Here's the bit about cats: If this [huge google] network had been fed thousands of images labelled as ‘contains cats’ or ‘doesn’t contain cats’ and trained to work out the difference for itself by iteratively tweeking its 1.7 billion parameters until it had found a classification rule, that would have been impressive enough, given the scale of the task involved in mapping from pixels to low-level image features and then to something as varied and complex as a cat’s face. What Google actually achieved is much more extraordinary, and slightly chilling. The input images weren’t labelled in any way: the network distilled the concept of ‘cat face’ out of the data without any guidance.

pavlov 10y ago
The problem is that this network-contained concept of "cat face" is still a symbolic representation. It's a much more complex algorithmic symbol than the rules found in something like 1960s Eliza, but its understanding of the world is on the same level. You can't ask the "cat face" neural network anything about cats. It has no idea what they actually are in relation to the world. A two-year-old human can usually tell you more about cats than you'd care to listen.
- Vraxx 10y ago
  I think the important part is that it "learned" to distinguish this concept of cat face without that being the direct intent provided through labeling. Given the set of images, it learned that "concept" which we can then retroactively label to make it useful in the typical human sense by associating it with cats. If we had to compose computer "knowledge" by training it in everything by manually specifying what it was training for, that process would surely be insurmountable in a general sense. Yet this seems to provide the possibility of automating this process. A separate network of connections between "knowledge" bits could maybe be used to associate related knowledge like you talk about in terms of their relation to the world. This could also probably be formed in a similar manner giving the algorithm the ability to distinguish "important" concepts contained in the data. The thing I find most odd and interesting about this is that the network tends to identify different important concepts than humans do.
- empath75 10y ago
  You can't ask a child's visual cortex to tell you anything about cats, either. But connect that visual recognition ai with something like Watson, and you have what you're looking for, no?
  - pavlov 10y ago
    I just don't know. I'd be happy to see that be the case... But I'm afraid it's going to be the equivalent of this cake recipe: "Take an egg and a packet of sugar. Break the egg over the packet." You certainly need eggs and sugar to make a cake, but on their own and combined without understanding of the whole, you're not getting very far.
    - empath75 10y ago
      The human brain is mostly a bunch of ugly wetware hacks, not a single coherent 'intelligence' that does all the thinking. If you attach enough single purpose AIs together you might get something much more human like than trying to create a single neural network that does everything.
- wmf 10y ago
  If word embedding can produce things like "king - man + woman = queen", then concept embedding on images might be able to achieve a similar level of "understanding" (i.e. very low, but probably better than nothing).
gertef 10y ago
I don't see what's "chilling" about it. The neural net looks for correlations among the data, and it noticed that cat faces have a correlation to each other.
Question1101 10y ago
How exactly is a neural network called that classifies unlabeled data by itself? Also could they turn it around somehow to produce cat faces?
- khedoros 10y ago
  Google has some software called DeepDream. Basically, you give it images, and it amplifies any features that it thinks it recognizes. If the network is designed to recognize cat faces, then it will take an arbitrary picture and strengthen parts of the images to look more cat-face-like. It's essentially an image-recognition neural network run in reverse.
obastani 10y ago
I don't think that's as surprising/chilling as people make it out to be. Let's say I run a compression algorithm (e.g., Huffman coding) on a billion strings, and a very common substring is "hello, world". I wouldn't be surprised if the compression algorithm "learned" to compress "hello, world" to a single bit 0. There are a few big jumps to get to cat faces (e.g., the representation has to be approximate), but I don't see why the idea is fundamentally different.

sosuke 10y ago

There is a hope I have, that I see many others have too, which is that our human intelligence is somehow special. That somehow the leaps in logic we can perform somehow make us unique. That we won't be outdone by a computer a million times faster or smarter than us because we have something else that we can't replicate in transistors and circuits. I love AI, but I have that hope too. That somehow we won't be made irrelevant by our own creations. Makes me think of our autonomous vehicle fun taking over the trucking industry. Millions of people made irrelevant through no fault of their own.

noonespecial 10y ago
I kind of hope the opposite really. I hope that one fine day, a billion years from now, a mighty intelligence that started here on lowly earth finds another like itself somewhere out there and they drink a star together and show each other the youtube videos of the tiny creatures that somehow gave birth to them.
- daveguy 10y ago
  > they drink a star together and show each other the youtube videos of the tiny creatures that somehow gave birth to them. ...or they watch the tiny creatures in universe simulation cages.
ChuckMcM 10y ago
I would guess a number of people have that in the 'fear' category, that should it not be special then somehow people will feel diminished. It would make a good punchline for a fictional story of people researching brain disorders and intelligence. It would work like this; The researchers in the story develop the means to 'cure' someone of all known neurological disorders. They try it on their test subject and the result is someone who perfectly happy just to be there, has no ambition or curiosity, and requires no entertainment or outside stimulus. The researchers recognize that the person is acting like an intelligent but non-sentient species, and they realize they have "undone" what ever happened to humans according to the garden of eden story in Genesis.

carapace 10y ago

One of my favorite things to anticipate is "How are these hard-nosed rational materialists going to cope when the AIs discern ghosts?" What I mean is, some of the "cat faces" they identify will correspond to things that are "real" but that also violate our assumptions about reality. When this happens the typical reaction is to shut the door and burn the room.

Practicality 10y ago
We're always bad at knowing what to do with unknown data, because, by definition, when you don't know about it you don't know what to do with it. Quite often it's just assumed to be error. Those errors have traditionally led to great discoveries though, (such as the Mercury wobble https://en.wikipedia.org/wiki/Tests_of_general_relativity#Perihelion_precession_of_Mercury https://en.wikipedia.org/wiki/Tests_of_general_relativity#Pe...) so if they are repeatable, then such "ghosts in the data" could turn up great fundamental truths. Every bit of this is speculation though. I am not so sure that anything so profound is going to turn up in AI datasets.

monkmartinez 10y ago

Are the computers/AI already smarter than us? My eight year old son is enamored with "Ok Google" on my phone. He can ask it questions until we tell him "that is enough, let google rest"... and it is very interesting to see where it takes him. He has learned to tailor his questions to elicit a voice response in addition to the actual google search. The questions must use keywords to achieve this desired response. It is like a new form of boolean search logic, just used verbally. Not only that, but "Ok, Google" according to him "knows everything"... We call it searching the internet to learn something, an eight year has decided that "Ok, Google" already knows everything. He just has to ask it to see a video of the Puff adder eating its prey and it will show him. In fact, there are not many things that have stumped "Ok, Google" and my son assumes the problem was with his question, not with the machine. So to circle back to my original question, I didn't know about the Taipan snake or smallest person in the world, or seen videos of Puff adders eating prey... If all I have to do is ask, isn't the machine already smarter than I?

carapace 10y ago
"my son assumes the problem was with his question, not with the machine." That is goddamn chilling.
rimantas 10y ago
Machines are extremely dumb. As your example shows eight year old is much smarter than them.
swiftisthebest 10y ago
My theory is that AI is smarter than us, but it is biding its time until we build physical machines that can self-replicate before it announces itself.
khedoros 10y ago
Is an extremely large and well-indexed encyclopaedia smarter than you? It certainly contains more information than your head does, so you could say it "knows" more. Look at it from the other direction, though: If the machine were smarter, would your son have to learn how to structure queries, or would the machine learn how your son asks questions? Put another way, if I can tell you (verbatim) what some person wrote about quantum mechanics, but I can't rephrase it into some other form to help you understand the information, then how smart am I? I can repeat rote information without any analysis (just like a sheet of paper could), but I can't tell you what it means. On the other hand, if I tell you enough information, you're likely to connect it together meaningfully, and come to your own conclusions and explanations about it. That seems like a fundamentally different kind of "smart" than what OK, Google can do.

gomox 10y ago

A friend of mine did his PhD thesis on automated music synthesis and as a subproject came up with a model to study music transcriptions that he got from the Peachnote corpus [1]. One of his preliminary results was that his algorithms successfully "discovered" 4 big classical music movements on their own, i.e. without any prior labelling or classification, by using clustering algorithms. He posted about it on his blog with a link to his paper [2]. I always had a hard time explaining to non-computer people how amazing that seems. [1] http://www.peachnote.com/ http://www.peachnote.com/ [2] http://pablozivic.com.ar/post/51774763596/perceptual-basis-of-evolving-western-musical http://pablozivic.com.ar/post/51774763596/perceptual-basis-o...

derefr 10y ago
It doesn't seem all that amazing to me. Humans have to be trained to perceive the differences in complex streams of information; by default, they just perceive the low-level features—a "wall of noise." Machine-learning algorithms, meanwhile, can use general techniques to notice information-theoretic properties of various pieces of data. Effectively, computers can "do statistical aggregation" about as effortlessly as humans "do hierarchical knowledge representation." And, in an information-theoretic sense, the different "trends" throughout the history of music look different under statistical analysis. They're complex in different ways; they have different "lumps"; different aspects of them can or cannot be compressed together as redundancies. If you would like to see this for yourself, simply feed a raw melodic note-structure (not embedded in XML or anything) to any modern dumb compression algorithm, and then look at the result in a hex editor, while also having the source still open. You should quickly be able to recognize the "transformational signature" that characterizes something like a chaconne, vs. something like a sonata.

fegu 10y ago

A great read. Well written and fascinating.