6 ms·

I'd love to hear Chomsky's reaction to this stuff (or someone in his camp on the Chomsky vs. Norvig debate [0]). My understanding is that Chomsky was against s

by feral 10y ago

I'd love to hear Chomsky's reaction to this stuff (or someone in his camp on the Chomsky vs. Norvig debate [0]). My understanding is that Chomsky was against statistical approaches to AI, as being scientifically un-useful - eventual dead ends, which would reach a certain accuracy, and plateau - as opposed to the purer logic/grammar approaches, which reductionistically/generatively decompose things into constituent parts, in some interpretable way, which is hence more scientifically valuable, and composable - easier to build on. But now we're seeing these very successful blended approaches, where you've got a grammatical search, which is reductionist, and produces an interpretable factoring of the sentence - but its guided by a massive (comparatively uninterpretable) neural net. It's like AlphaGo - which is still doing search, in a very structured, rule based, reductionist way - but leveraging the more black-box statistical neural network to make the search actually efficient, and qualitatively more useful. Is this an emerging paradigm? I used to have a lot of sympathy for the Compsky argument, and thought Norvig et al. [the machine learning community] could be accused of talking up a more prosaic 'applied ML' agenda into being more scientifically worthwhile than it actually was. But I think systems like this are evidence that gradual, incremental, improvement of working statistical systems, can eventually yield more powerful reductionist/logical systems overall. I'd love to hear an opposing perspective from someone in the Chomsky camp, in the context of systems like this. (Which I am hopefully not strawmanning here.) [0]Norvig's article: http://norvig.com/chomsky.html http://norvig.com/chomsky.html

x5n1 10y ago
I think you are right and I think in the human brain similar sort of hybrid processes happen in order to make sense of the world. In the end strong AI will look very much like a massive hybrid system and a conscious controller that takes and integrates that information into a understood model of the world.
DonaldFisk 10y ago
I largely agree with Chomsky. I think both approaches are needed for general AI: neural networks, or something like them, for low level perception and recognition; and symbolic AI for higher level reasoning. Without the symbolic layer, you can't be sure what's going on. Symbolic AI has been very closely guided by cognitive psychology. Artificial neural networks ignore neurophysiology, so even when they work, they tell us very little about how the brain works. I keep hearing claims that symbolic AI is the wrong approach for anything, and that it failed. Yet there were quite a few successes (expert systems, discovery learning, common sense reasoning, for example) before sources of funding dried up.
- nl 10y ago
  Artificial neural networks ignore neurophysiology, so even when they work, they tell us very little about how the brain works. That is completely wrong. People like Geoff Hinton spend most of their time thinking about how the brain works (indeed, his background is cognitive psychology). The "convolution" part of convolution neural networks is designed to mimic how the optic nerve interfaces with the brain. I keep hearing claims that symbolic AI is the wrong approach for anything, and that it failed. Yet there were quite a few successes (expert systems, discovery learning, common sense reasoning, for example) before sources of funding dried up. The funding dried up because they ran into the limits of what is possible.
  - DonaldFisk 10y ago
    No equivalent of error backpropagation has ever been found in real neurons, and it's biologically implausible. So ANNs are almost certainly using a different learning mechanism from the one used in the brain. Even single neurons are quite complex and very little of this complexity is present in neural networks. The visual system (retina, lateral geniculate nucleus, visual cortex) was fairly well understood well before ANNs were developed. A few uncontroversial ideas (e.g. that cells take their inputs from neighbouring cells in the previous layer) were adopted for use in ANNs. I was around at the time of, and affected by, the AI winter. There was certainly no consensus among those working in AI that they had got as far as they could. Work stopped when funding was cut, often for political reasons. The most mature area at the time, apparently ripe for commercialization, was expert systems. However, it was very hard to commercialize them: customers couldn't think of any suitable applications, and when they could, they couldn't spare the time of their experts. Finally, the main reason for the AI winter was probably that AI was unable to live up to the grossly inflated expectations, simply because the expectations were grossly inflated. This seems to be happening again, with neural networks.
    - colllectorof 10y ago
      > I was around at the time of, and affected by, the AI winter. There was certainly no consensus among those working in AI that they had got as far as they could. Work stopped when funding was cut, often for political reasons. I wasn't around, but I got curious about symbolic systems after listening to MIT's AI course[1]. Did some reading about the subject. The impression I got matches what you describe. It's ridiculous how many people here dogmatically recite statements about failures of symbolic systems without (apparently) knowing anything about how those systems were used and what they achieved. If you listen to the comments, it sounds as if research on symbolic systems only ever produced crude, useless toys. That was certainly my impression before I took some time to actually look into it. A bit of straightforward Googling can show that it's a gross misrepresentation of history. For example, MIT's lecture on knowledge engineering [2] has some really interesting info on this subject. [1] http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-034-artificial-intelligence-fall-2010/index.htm http://ocw.mit.edu/courses/electrical-engineering-and-comput... [2] http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-871-knowledge-based-applications-systems-spring-2005/index.htm http://ocw.mit.edu/courses/electrical-engineering-and-comput...
fighting 10y ago
Not in either camp, quit the nlp field due to disillusionment that it would lead to anything useful or meaningful. Both rule-based and statistical approaches are fundamentally flawed by not incorporating any real world information. Humans do language going from real world info, mapping to grammar or rules. Computers are trying to go the other way and are not going to succeed other than as mere toys. Even tech progression-wise, both rule and model/nn approaches are really bad since there is no meaningful sense of iterative progress or getting better step by step, unlike cpu chips or memory speeds. They are more of a random search in a vast space, hit or miss, getting lucky or not, which is very bad no good, as a technology or as a career.
- igravious 10y ago
  How about looking at it this way. We now have multiple implementations that get 94%+ success _without_ knowing anything about the world. Isn't that remarkable? Now to get to 99.4%+ how about we combine techniques such as spaCy or Parsey McParseFace (love the name Google) with very simple real-world cognitive models. So for the example given "Alice drove down the street in her car." a simple cognitive model would _know_ that streets cannot be in cars and so be able to disambiguate. A cognitive model wouldn't know all the facts about the world, it would know certain things about streets, certain things about cars and be able to infer on the fly whether the relationship between streets and cars matches either the first parse possibility or the second. To me this seems like the obvious next step. If it's obvious to me it must have been obvious to someone else so presumably somebody is working on it.
  - fighting 10y ago
    The 94% success rate is in made up, limited tests. Real world, they fail constantly in weird horrific or laughable ways. See any Microsoft AI public demo ever. It's like the self driving car claim of millions of miles without accidents, except that humans took over anytime there was a chance of one.
  - adrianratnapala 10y ago
    I suspect another way of looking at it is that these models actually learn about the real world by reading about it in the WSJ -- of course their knowledge of it is not as deep as our own, but good enough for what they do. That is, if you took the well NLP trained model, then you could in principle extract out of it facts like "streets are not found inside cars".
unhammer 10y ago
I don't think you're strawmanning in general, there have been a lot of symbolic AI people who scoffed at any mention of statistics or real-world data, but it's not the case that you have to eschew all empiricism just because you use rules. See e.g. http://visl.sdu.dk/~eckhard/pdf/TIL2006.pdf http://visl.sdu.dk/~eckhard/pdf/TIL2006.pdf which gets 99% on POS and 96% on syntax function assignment – Constraint Grammar parsers are the state of the art of rule-based systems, and the well-developed ones beat statistical systems. CG's are also multitaggers – they don't assume a word has to have only one reading, it might actually be ambiguous, and in that case it shouldn't be further disambiguated (that's why they use F-scores instead of plain "accuracy"). CG's also require manual work, so it's not like you can download a corpus an unsupervisedly learn everything; but on the other hand, for what languages in the world do you have a large enough data set to unsupervisedly learn a good model? And for what training methods can you even get good models from unlabeled data? The set of languages for which there are large annotated corpora (especially treebanks) is even smaller … So CG's are also heavily used for lesser-resourced languages (typically in combination with finite state transducers for morphological analysis), where the lack of training data means it's a lot more cost-effective to write rules (and turn existing dictionaries into machine-readable FST's) than it is to create annotated training data (which would often involve OCR-ing texts, introducing yet another error source). CG writers still tend to have a very empirical mindset – no toy sentences like "put the cone on the block", but continual testing on any real-world text they can get their hands on.
YeGoblynQueenne 10y ago
Try writing to Chomsky. He's well known for replying personally to his email. I know that for a fact, he replied to one of mine once (not about language )
- andreasvc 10y ago
  Please don't. If you're going to bother the man, ask a more interesting question. He's made his position on this abundantly clear in the Chomsky-Norvig debate.
foldr 10y ago
Chomsky was never really interested in AI at all. I don't see anything in these results that has any implications for any position that Chomsky's taken. Chomsky's always pretty much taken it for granted that surface constituency structure can be extracted by statistical methods.

by feral 10y ago

x5n1 10y ago

I think you are right and I think in the human brain similar sort of hybrid processes happen in order to make sense of the world. In the end strong AI will look very much like a massive hybrid system and a conscious controller that takes and integrates that information into a understood model of the world.

DonaldFisk 10y ago

I largely agree with Chomsky. I think both approaches are needed for general AI: neural networks, or something like them, for low level perception and recognition; and symbolic AI for higher level reasoning. Without the symbolic layer, you can't be sure what's going on. Symbolic AI has been very closely guided by cognitive psychology. Artificial neural networks ignore neurophysiology, so even when they work, they tell us very little about how the brain works. I keep hearing claims that symbolic AI is the wrong approach for anything, and that it failed. Yet there were quite a few successes (expert systems, discovery learning, common sense reasoning, for example) before sources of funding dried up.

nl 10y ago
Artificial neural networks ignore neurophysiology, so even when they work, they tell us very little about how the brain works. That is completely wrong. People like Geoff Hinton spend most of their time thinking about how the brain works (indeed, his background is cognitive psychology). The "convolution" part of convolution neural networks is designed to mimic how the optic nerve interfaces with the brain. I keep hearing claims that symbolic AI is the wrong approach for anything, and that it failed. Yet there were quite a few successes (expert systems, discovery learning, common sense reasoning, for example) before sources of funding dried up. The funding dried up because they ran into the limits of what is possible.
- DonaldFisk 10y ago
  No equivalent of error backpropagation has ever been found in real neurons, and it's biologically implausible. So ANNs are almost certainly using a different learning mechanism from the one used in the brain. Even single neurons are quite complex and very little of this complexity is present in neural networks. The visual system (retina, lateral geniculate nucleus, visual cortex) was fairly well understood well before ANNs were developed. A few uncontroversial ideas (e.g. that cells take their inputs from neighbouring cells in the previous layer) were adopted for use in ANNs. I was around at the time of, and affected by, the AI winter. There was certainly no consensus among those working in AI that they had got as far as they could. Work stopped when funding was cut, often for political reasons. The most mature area at the time, apparently ripe for commercialization, was expert systems. However, it was very hard to commercialize them: customers couldn't think of any suitable applications, and when they could, they couldn't spare the time of their experts. Finally, the main reason for the AI winter was probably that AI was unable to live up to the grossly inflated expectations, simply because the expectations were grossly inflated. This seems to be happening again, with neural networks.
  - colllectorof 10y ago
    > I was around at the time of, and affected by, the AI winter. There was certainly no consensus among those working in AI that they had got as far as they could. Work stopped when funding was cut, often for political reasons. I wasn't around, but I got curious about symbolic systems after listening to MIT's AI course[1]. Did some reading about the subject. The impression I got matches what you describe. It's ridiculous how many people here dogmatically recite statements about failures of symbolic systems without (apparently) knowing anything about how those systems were used and what they achieved. If you listen to the comments, it sounds as if research on symbolic systems only ever produced crude, useless toys. That was certainly my impression before I took some time to actually look into it. A bit of straightforward Googling can show that it's a gross misrepresentation of history. For example, MIT's lecture on knowledge engineering [2] has some really interesting info on this subject. [1] http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-034-artificial-intelligence-fall-2010/index.htm http://ocw.mit.edu/courses/electrical-engineering-and-comput... [2] http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-871-knowledge-based-applications-systems-spring-2005/index.htm http://ocw.mit.edu/courses/electrical-engineering-and-comput...

fighting 10y ago

Not in either camp, quit the nlp field due to disillusionment that it would lead to anything useful or meaningful. Both rule-based and statistical approaches are fundamentally flawed by not incorporating any real world information. Humans do language going from real world info, mapping to grammar or rules. Computers are trying to go the other way and are not going to succeed other than as mere toys. Even tech progression-wise, both rule and model/nn approaches are really bad since there is no meaningful sense of iterative progress or getting better step by step, unlike cpu chips or memory speeds. They are more of a random search in a vast space, hit or miss, getting lucky or not, which is very bad no good, as a technology or as a career.

igravious 10y ago
How about looking at it this way. We now have multiple implementations that get 94%+ success _without_ knowing anything about the world. Isn't that remarkable? Now to get to 99.4%+ how about we combine techniques such as spaCy or Parsey McParseFace (love the name Google) with very simple real-world cognitive models. So for the example given "Alice drove down the street in her car." a simple cognitive model would _know_ that streets cannot be in cars and so be able to disambiguate. A cognitive model wouldn't know all the facts about the world, it would know certain things about streets, certain things about cars and be able to infer on the fly whether the relationship between streets and cars matches either the first parse possibility or the second. To me this seems like the obvious next step. If it's obvious to me it must have been obvious to someone else so presumably somebody is working on it.
- fighting 10y ago
  The 94% success rate is in made up, limited tests. Real world, they fail constantly in weird horrific or laughable ways. See any Microsoft AI public demo ever. It's like the self driving car claim of millions of miles without accidents, except that humans took over anytime there was a chance of one.
- adrianratnapala 10y ago
  I suspect another way of looking at it is that these models actually learn about the real world by reading about it in the WSJ -- of course their knowledge of it is not as deep as our own, but good enough for what they do. That is, if you took the well NLP trained model, then you could in principle extract out of it facts like "streets are not found inside cars".

unhammer 10y ago

I don't think you're strawmanning in general, there have been a lot of symbolic AI people who scoffed at any mention of statistics or real-world data, but it's not the case that you have to eschew all empiricism just because you use rules. See e.g. http://visl.sdu.dk/~eckhard/pdf/TIL2006.pdf http://visl.sdu.dk/~eckhard/pdf/TIL2006.pdf which gets 99% on POS and 96% on syntax function assignment – Constraint Grammar parsers are the state of the art of rule-based systems, and the well-developed ones beat statistical systems. CG's are also multitaggers – they don't assume a word has to have only one reading, it might actually be ambiguous, and in that case it shouldn't be further disambiguated (that's why they use F-scores instead of plain "accuracy"). CG's also require manual work, so it's not like you can download a corpus an unsupervisedly learn everything; but on the other hand, for what languages in the world do you have a large enough data set to unsupervisedly learn a good model? And for what training methods can you even get good models from unlabeled data? The set of languages for which there are large annotated corpora (especially treebanks) is even smaller … So CG's are also heavily used for lesser-resourced languages (typically in combination with finite state transducers for morphological analysis), where the lack of training data means it's a lot more cost-effective to write rules (and turn existing dictionaries into machine-readable FST's) than it is to create annotated training data (which would often involve OCR-ing texts, introducing yet another error source). CG writers still tend to have a very empirical mindset – no toy sentences like "put the cone on the block", but continual testing on any real-world text they can get their hands on.

YeGoblynQueenne 10y ago

Try writing to Chomsky. He's well known for replying personally to his email. I know that for a fact, he replied to one of mine once (not about language )

andreasvc 10y ago
Please don't. If you're going to bother the man, ask a more interesting question. He's made his position on this abundantly clear in the Chomsky-Norvig debate.

foldr 10y ago

Chomsky was never really interested in AI at all. I don't see anything in these results that has any implications for any position that Chomsky's taken. Chomsky's always pretty much taken it for granted that surface constituency structure can be extracted by statistical methods.