11 ms·

On Chomsky and the Two Cultures of Statistical Learning

313 points by EdiX 15y ago

CWuestefeld 15y ago
Server's down. Here's a cached link: http://webcache.googleusercontent.com/search?q=cache:http%3A%2F%2Fnorvig.com%2Fchomsky.html http://webcache.googleusercontent.com/search?q=cache:http%3A... EDIT: stop giving me upvotes. I've got 11 points now for nothing more than a link. I don't deserve them. Stupid hidden points...
- jng 15y ago
  People upvote so that the link to the cached copy will be at the top.
- norvig 15y ago
  Sorry about the intermittent access. My hosting service provides me with sufficient bandwidth, but only provides a version of Apache that forks a new process for every GET, and thus runs out of processes and denies access to a portion of visitors when I get slashdotted/redditted/hacker-newsified. If anyone can suggest a more reasonable hosting service, let me know. -Peter Norvig
  - deleted 15y ago
    [deleted]
  - alphamerik 15y ago
    [cough] I hear Google has pretty good bandwidth and scaling. Ever try App Engine? [/cough]
PaulHoule 15y ago
It's funny. Lately I've been working with NLP systems and in the last few years there are a few really good parts-of-speech taggers that are about 99% accurate. All the ones I know of are based on hidden markov models, which definitely would disappoint Chomsky. Part of the trouble w/ Chomsky is that real language doesn't draw a clear line between syntax and semantics. Even though an HMM doesn't correctly model the nested structures that are common in natural language, it makes up for it by encoding semantic information.
- sharmajai 15y ago
  Another trouble is that human beings are innately probabilistic when it comes to language. A sentence written/spoken by humans does not have to be gramatically correct, to convey it's meaning, and does not always follow the strict rules that Chomsky talks about. It's not the language that defines how we communicate, it's how we communicate defines the language. But I also disagree with peter when he says the why is not important, it is this why or the understanding of the matter that separates us from the machines like watson, since our sole purpose in life is not to win at a game, but play/enjoy the game and most importantly "reuse the understanding" gained in some other facet of life, a feat that I beleive no machine is capable of.
- foldr 15y ago
  >It's funny. Lately I've been working with NLP systems and in the last few years there are a few really good parts-of-speech taggers that are about 99% accurate. All the ones I know of are based on hidden markov models, which definitely would disappoint Chomsky. No, it wouldn't disappoint him at all. In fact, one of his earliest works in linguistics discussed how transition probabilities could be used for chunking and categorization. (See http://www.tilburguniversity.edu/research/institutes-and-research-groups/tilps/philli/chomsky.pdf http://www.tilburguniversity.edu/research/institutes-and-res... ) It's not as if Chomsky ever presented part of speech tagging as a poverty of the stimulus argument.
wccrawford 15y ago
"O'Reilly is correct that these questions can only be addressed by mythmaking, religion or philosophy, not by science." ... My jaw is on the floor. It drives me nuts when people go from 'We can't explain that yet' to 'The only explanation is God.' The tides are incredibly complex when you insist on 'why' all the way back to the beginning of the universe. Everything is!
- paganel 15y ago
  > My jaw is on the floor. It drives me nuts when people go from 'We can't explain that yet' to 'The only explanation is God.' I don't think anyone mentioned God, and, to be fair, the problem of what exactly constitutes reality and what would be the best ways to imitate it is quite complex. We, as a species, have been trying to find the rational answer to this problem for at least 2,500 years (since the pre-Socratics), but as far as I know we haven't come to any definitive answer, we don't even know if there is such an answer.
  - ugh 15y ago
    And why exactly are myths, religions or philosophy better at answering questions than science? What’s the justification for that claim?
    - paganel 15y ago
      > And why exactly are myths, religions or philosophy better at answering questions than science? What’s the justification for that claim? Science assumes that there is a "justification" (for language, the universe, you name it). A true philosopher always begins by asking himself if there is such a thing as "justification". Just think about it, science has helped us put a man on the Moon and create the Tsar bomb, but it isn't able to answer Epimenides's "All Cretans are liars" paradox, a 2,500 years-old problem.
      - ugh 15y ago
        What’s there to answer? It’s a paradox.
christianpbrink 15y ago
"If Chomsky had focused on the other side, interpretation, as Claude Shannon did, he may have changed his tune. In interpretation (such as speech recognition) the listener receives a noisy, ambiguous signal and needs to decide which of many possible intended messages is most likely. Thus, it is obvious that this is inherently a probabilistic problem, as was recognized early on by all researchers in speech recognition..." This is the money shot especially since speakers are aware of the interpretive activity of listeners, and effective speakers play constantly on the ambiguities in their statements - structural (i.e. grammatical) ambiguities as well as semantic ambiguities. Listeners in turn are aware of speakers' awareness of this.. There is, effectively, an infinity of mutual awarenesses of structural ambiguities. In any instance of communication. I think most technologists and (especially) businesspeople see this intuitively. I think many academics do not. Not sure how to articulate what I mean but I think I am saying something non-trivial about academics and their perspective on language.
- cma 15y ago
  Freeman Dyson earlier this year on this type of ambiguity as expressed in the drum language of the Democratic Republic of Congo: http://www.nybooks.com/articles/archives/2011/mar/10/how-we-know/ http://www.nybooks.com/articles/archives/2011/mar/10/how-we-...
- bluekeybox 15y ago
  > I think I am saying something non-trivial about academics and their perspective on language. I became convinced that there is a strain of thought, one that is especially pervasive in the academia, which believes that knowledge/meaning is something irreducible and almost mystical. It probably has to do with the fact that people who fetishize knowledge as something incredibly worthwhile for its own sake end up being overrepresented in the academia. Those who are a bit more cynical/nihilistic tend to go into finance or start their own companies. The old advice "do not make any gods to be alongside me" is still relevant except for the "alongside me" part, which probably only has any meaning if you consider yourself religious. I have a feeling that many academicians, especially the old-school ones, idolize knowledge to the extent of ascribing to it god-like powers even if said knowledge has little relevance for anything practical.
cma 15y ago
Chomsky's April 8th lecture at Carelton University on language had several thoughts on machine translation: http://www.youtube.com/watch?v=XbjVMq0k3uc http://www.youtube.com/watch?v=XbjVMq0k3uc (I think it even had the same bee-dance example)
brockf 15y ago
Chomsky's one paragraph quote at the beginning of this article is more clear and thoughtful than the rest of this. I feel the author's missing the point. In the case of language, observing and reporting statistical probabilities in written/spoken language output does very little to explain the cognitive systems used in acquiring and using language. Even one statistical anomaly serves to show that statistical learning is NOT the entire picture when it comes to language development. There was another article on HN a while back that had another great quote from Chomsky that does well to illustrate what I feel is his main point here: "Fooling people into mistaking a submarine for a whale doesn't show that submarines really swim; nor does it fail to establish the fact". Creating a computer that can produce millions of grammatical utterances does little to show that we understand language systems. Now, if a computer could - like humans - learn to produce infinite, novel, contextual, and meaningful grammatical utterances, that's a different story. But that story will take a lot more than statistical learning to write.
- andrewcooke 15y ago
  just because you only understand one side of an argument doesn't mean everyone else is an idiot. in some sense this is the same argument as searle's chinese room. that's sufficiently well known and debated that it's fair to say that neither side can be dismissed as simply "missing the point".
  - brockf 15y ago
    Throughout history, there have been many "well known and debated" arguments that have proven to be idiotic. See: the shape of the Earth. This isn't one of them, though. And I didn't mean to imply that anyone was an idiot. Instead of critiquing how I said something, or that I said something at all, can you tell me what I am missing? I'm obviously missing something - Norvig is no idiot.
    - andrewcooke 15y ago
      the problem is whether or not there is any way to "ground" meaning. for physics, the "unreasonable effectiveness of mathematics" might suggest that there are simple "meanings" that underly physical "laws". but there's nothing to say that the same is true for intelligence or language. maybe the brain is nothing more than a particularly flexible "neural net", which statistical methods are modelling quite well. in that case, "intelligence" is not qualitatively different from "a good simulation of intelligence". the same problem occurs in free will - does it "really" exist? if we're just (mechanical, predictable, although highly complex) machines then it is difficult to imagine how it can. yet the intuition is that there is clearly some meaning to the idea of a "free agent". these are hard questions. people don't know the answers. instead we look to what daniel dennett calls "intuition pumps" (see his book "elbow room" on the free will problem) - simple parallels that "feel right". from those, we use intuition to argue in one direction or another. but the problem with that approach is that it depends on what you choose as a "hint". some advance is being made through experiment. imaging of neural activity in the brain, for example, or the recent discovery that people who believe they have free will behave differently to those that don't. searle's chinese room argument - http://plato.stanford.edu/entries/chinese-room/ http://plato.stanford.edu/entries/chinese-room/ unreasonable effectiveness of mathematics - http://www.jstor.org/pss/2321982 http://www.jstor.org/pss/2321982 http://plato.stanford.edu/entries/mathematics-explanation/ http://plato.stanford.edu/entries/mathematics-explanation/ i can't find the free will behaviour result that was in the news about a week ago. but that's perhaps less related to this anyway. and i doubt that everyone who thought the earth was flat was idiotic, frankly. just because something is obvious now doesn't mean it was a stupid question, or easy to answer, when first raised.
sethg 15y ago
This essay made me think: Lojban (http://www.lojban.org/tiki/la+lojban.+mo http://www.lojban.org/tiki/la+lojban.+mo), among constructed languages, is the categorial language par excellence. Every word has a well-defined range of meaning; the grammar can be parsed by the same kinds of parsers used for programming languages; potential sources of ambiguity, like plural references, associativity of modifiers, and negation, have been rigorously (or tediously, depending on how you roll) nailed down. Can there be such a thing as a conlang that demonstrates the ideal statistical grammar and semantics? (“All the words in this list are 60% likely to be used as nouns and 40% likely to be used as verbs....” But in the absence of a pre-existing linguistic community, how could you get students of the language to use them in the right proportions?)
borism 15y ago
And while it may seem crass and anti-intellectual to consider a financial measure of success Why are other metrics Norvig provides like articles published or prevalence in practical applications are considered more intellectual? And besides, I don't think "accurately modeling the world" is the end of it. Classical Newtonian mechanics correctly describe 99% of our activities in the real world and were considered pinnacle of scientific achievement for several centuries. Yet we know today that they're just a subset of General relativity and Quantum mechanics.
Jun8 15y ago
This is not a new debate. Within Linguistics there has been a continuous push against statistical NLP models. Read the introduction of Manning's book, even he seems to be defensive about NLP. Chomsky is a colossus, his achievements are well-known. However, at one point in many disciplines it comes to pass that the pioneers who pave the way in time become the very impediment to new ideas. His emphasis on Semantics have warped the minds of many generations of researchers (and some other ideas on universal grammar, too). I experienced this first hand, my advisor, Prof. Raskin, a great researcher on semantics, nevertheless thought that statistical approaches were not the way to go. Sadly, in many Linguistics departments people are just not equipped with the statistical tools necessary to have a basic understand of what's being done in the NLP field. So NLP is generally taught under CS, EE, or CompE.
- adavies42 15y ago
  i saw someone once compare chomsky to freud, as a foundational figure whose discipline can't/couldn't progress during his lifetime.
  - ordinary 15y ago
    Einstein would be another example.
- foldr 15y ago
  >His emphasis on Semantics What are you talking about? Chomsky has always been highly critical of formal semantics.
noahlt 15y ago
Strangely appropriate is today's XKCD: http://xkcd.com/904/ http://xkcd.com/904/
- kenjackson 15y ago
  Hmm... I never thought of it that way. That sports are a weighted random number generator,but the various weights are unknown. And the commentators are discussing theories as to what the weights are, and how derived. (Although the cartoon seems to be saying the narratives are just about the numbers generated, which is more cynical, and frankly less interesting).
  - yourcelf 15y ago
    Actually, Larry Birnbaum over at Northwestern is doing exactly that: http://infolab.northwestern.edu/projects/stats-monkey/ http://infolab.northwestern.edu/projects/stats-monkey/ They take the coded sports results, and automatically generate narratives using statistical speech models. They have a startup that is doing it too, don't recall the name of it.... EDIT: I believe this is it: http://narrativescience.com/ http://narrativescience.com/
T_S_ 15y ago
The handshake example was illuminating. Three "equivalent" theories: Theory A: Closed form formula function. Theory B: "Algorithm". Still a function. Theory C: Memoized function (constant time!) According to the article "nobody" likes C, especially the article's Chomsky straw man. If one had a procedure to convert C to A, then this whole issue would become hairsplitting. Such a procedure would aim to convert a memoized function back into a form that uses more symbols from a mathematical language. A good criteria of success would be the description length of the resulting procedure in the preferred language. One reason this could be useful to science is that once you identify a value that is useful in many theories it becomes part of the language. Making it available to the next problem may speed up the search for a "good" description of the next phenomenon. Identical procedures that appeared in various algorithms might acquire a special name. One such value might be called "pi", another "foldr" and so on. Of course there may be many good descriptions, just as there are many languages. Also, the example could be extended to statistical modeling situations by adding room for error terms in the suitability criteria. So if, you have a general procedure to convert a table into a definition you can make money and science at the same time!
niels_olson 15y ago
This whole theory vs observation argument exists at the very pinnacle of human thought, expressed in the Copenhagen interpretation. If you want to contribute to the human understanding of this, you'll have to beat Bohr and the uncertainty principle.
- Create 15y ago
  Fourier was there first.
  - niels_olson 15y ago
    My claim wasn't first, it was top. You up-end the Copenhagen interpretation, show the universe really is deterministic, and every other argument on this subject, in every discipline, collapses. As it is, the arguments are almost certainly failed, but it's not quite a cinch. Because probability admits determinism as a special case. One of the deep points of Norvig's essay.
double-z 15y ago
The commentary has nothing to do with what Chomsky proposed. The author defines success as "being successful at accomplishing a task". That has nothing to do with science. Full stop.
macmac 15y ago
Are Norvig's comments on the "I before E except before C." really vaild? Why would one use a corpus for analysis of the rule, and not a dictionary? It appears to me that "CIE" (P(CIE) = 0.0014) is more common than "CEI" (P(CEI) = 0.0005) because the words that contain the "exception" "CIE" are used more frequently in the corpus than the words that follow the rule "CEI". Once you know the limited number of exceptions (in the dictionary sense) the rule appears to preserve its relevance.
- jimbokun 15y ago
  I suppose a the most useful corpus for this rule would be spelling tests.
stcredzero 15y ago
My conclusion is that 100% of these articles are more about "accurately modeling the world" then they are about "providing insight," although they all have some theoretical insight component as well. Before you can figure out why, you have to make sure you can accurately characterize the what. So there's a lot of science that is focused on coming up with a descriptive tool like an adhoc curve, before the underlying principles are discovered. I think Chomsky is afraid that statistical models will cause people to stop looking for the underlying principles.
davidmathers 15y ago
Chomsky called the Watson computer that won Jeopardy "a bigger bulldozer." He goes into more detail about his AI opinions here: http://www.framingbusiness.net/archives/1366 http://www.framingbusiness.net/archives/1366
galactus 15y ago
It is interesting than on a completely different debate, chomsky takes norvig's position (he is accused of not looking for a "theory" and "whys" and he replies that it is pragmatic results that matter): http://mindfulpleasures.blogspot.com/2011/01/noam-chomsky-on-derrida-foucault-lacan.html http://mindfulpleasures.blogspot.com/2011/01/noam-chomsky-on...
_grrr 15y ago
I've been monitoring the page this post points to with a bookmarking tool we've just released in beta. Here are the latest set of changes: http://app.bookmarkerpro.com/changes?fmt=html&id=2573 http://app.bookmarkerpro.com/changes?fmt=html&id=2573 Quite a few revisions since first posted to HN!
- _grrr 15y ago
  More revisions... http://tinyurl.com/3sabdc9 http://tinyurl.com/3sabdc9

313 points by EdiX 15y ago

Server's down. Here's a cached link: http://webcache.googleusercontent.com/search?q=cache:http%3A%2F%2Fnorvig.com%2Fchomsky.html http://webcache.googleusercontent.com/search?q=cache:http%3A... EDIT: stop giving me upvotes. I've got 11 points now for nothing more than a link. I don't deserve them. Stupid hidden points...

jng 15y ago
People upvote so that the link to the cached copy will be at the top.
norvig 15y ago
Sorry about the intermittent access. My hosting service provides me with sufficient bandwidth, but only provides a version of Apache that forks a new process for every GET, and thus runs out of processes and denies access to a portion of visitors when I get slashdotted/redditted/hacker-newsified. If anyone can suggest a more reasonable hosting service, let me know. -Peter Norvig
- deleted 15y ago
  [deleted]
- alphamerik 15y ago
  [cough] I hear Google has pretty good bandwidth and scaling. Ever try App Engine? [/cough]

PaulHoule 15y ago

It's funny. Lately I've been working with NLP systems and in the last few years there are a few really good parts-of-speech taggers that are about 99% accurate. All the ones I know of are based on hidden markov models, which definitely would disappoint Chomsky. Part of the trouble w/ Chomsky is that real language doesn't draw a clear line between syntax and semantics. Even though an HMM doesn't correctly model the nested structures that are common in natural language, it makes up for it by encoding semantic information.

sharmajai 15y ago
Another trouble is that human beings are innately probabilistic when it comes to language. A sentence written/spoken by humans does not have to be gramatically correct, to convey it's meaning, and does not always follow the strict rules that Chomsky talks about. It's not the language that defines how we communicate, it's how we communicate defines the language. But I also disagree with peter when he says the why is not important, it is this why or the understanding of the matter that separates us from the machines like watson, since our sole purpose in life is not to win at a game, but play/enjoy the game and most importantly "reuse the understanding" gained in some other facet of life, a feat that I beleive no machine is capable of.
foldr 15y ago
>It's funny. Lately I've been working with NLP systems and in the last few years there are a few really good parts-of-speech taggers that are about 99% accurate. All the ones I know of are based on hidden markov models, which definitely would disappoint Chomsky. No, it wouldn't disappoint him at all. In fact, one of his earliest works in linguistics discussed how transition probabilities could be used for chunking and categorization. (See http://www.tilburguniversity.edu/research/institutes-and-research-groups/tilps/philli/chomsky.pdf http://www.tilburguniversity.edu/research/institutes-and-res... ) It's not as if Chomsky ever presented part of speech tagging as a poverty of the stimulus argument.

wccrawford 15y ago

"O'Reilly is correct that these questions can only be addressed by mythmaking, religion or philosophy, not by science." ... My jaw is on the floor. It drives me nuts when people go from 'We can't explain that yet' to 'The only explanation is God.' The tides are incredibly complex when you insist on 'why' all the way back to the beginning of the universe. Everything is!

paganel 15y ago
> My jaw is on the floor. It drives me nuts when people go from 'We can't explain that yet' to 'The only explanation is God.' I don't think anyone mentioned God, and, to be fair, the problem of what exactly constitutes reality and what would be the best ways to imitate it is quite complex. We, as a species, have been trying to find the rational answer to this problem for at least 2,500 years (since the pre-Socratics), but as far as I know we haven't come to any definitive answer, we don't even know if there is such an answer.
- ugh 15y ago
  And why exactly are myths, religions or philosophy better at answering questions than science? What’s the justification for that claim?
  - paganel 15y ago
    > And why exactly are myths, religions or philosophy better at answering questions than science? What’s the justification for that claim? Science assumes that there is a "justification" (for language, the universe, you name it). A true philosopher always begins by asking himself if there is such a thing as "justification". Just think about it, science has helped us put a man on the Moon and create the Tsar bomb, but it isn't able to answer Epimenides's "All Cretans are liars" paradox, a 2,500 years-old problem.
    - ugh 15y ago
      What’s there to answer? It’s a paradox.

christianpbrink 15y ago

"If Chomsky had focused on the other side, interpretation, as Claude Shannon did, he may have changed his tune. In interpretation (such as speech recognition) the listener receives a noisy, ambiguous signal and needs to decide which of many possible intended messages is most likely. Thus, it is obvious that this is inherently a probabilistic problem, as was recognized early on by all researchers in speech recognition..." This is the money shot especially since speakers are aware of the interpretive activity of listeners, and effective speakers play constantly on the ambiguities in their statements - structural (i.e. grammatical) ambiguities as well as semantic ambiguities. Listeners in turn are aware of speakers' awareness of this.. There is, effectively, an infinity of mutual awarenesses of structural ambiguities. In any instance of communication. I think most technologists and (especially) businesspeople see this intuitively. I think many academics do not. Not sure how to articulate what I mean but I think I am saying something non-trivial about academics and their perspective on language.

cma 15y ago
Freeman Dyson earlier this year on this type of ambiguity as expressed in the drum language of the Democratic Republic of Congo: http://www.nybooks.com/articles/archives/2011/mar/10/how-we-know/ http://www.nybooks.com/articles/archives/2011/mar/10/how-we-...
bluekeybox 15y ago
> I think I am saying something non-trivial about academics and their perspective on language. I became convinced that there is a strain of thought, one that is especially pervasive in the academia, which believes that knowledge/meaning is something irreducible and almost mystical. It probably has to do with the fact that people who fetishize knowledge as something incredibly worthwhile for its own sake end up being overrepresented in the academia. Those who are a bit more cynical/nihilistic tend to go into finance or start their own companies. The old advice "do not make any gods to be alongside me" is still relevant except for the "alongside me" part, which probably only has any meaning if you consider yourself religious. I have a feeling that many academicians, especially the old-school ones, idolize knowledge to the extent of ascribing to it god-like powers even if said knowledge has little relevance for anything practical.

cma 15y ago

Chomsky's April 8th lecture at Carelton University on language had several thoughts on machine translation: http://www.youtube.com/watch?v=XbjVMq0k3uc http://www.youtube.com/watch?v=XbjVMq0k3uc (I think it even had the same bee-dance example)

brockf 15y ago

Chomsky's one paragraph quote at the beginning of this article is more clear and thoughtful than the rest of this. I feel the author's missing the point. In the case of language, observing and reporting statistical probabilities in written/spoken language output does very little to explain the cognitive systems used in acquiring and using language. Even one statistical anomaly serves to show that statistical learning is NOT the entire picture when it comes to language development. There was another article on HN a while back that had another great quote from Chomsky that does well to illustrate what I feel is his main point here: "Fooling people into mistaking a submarine for a whale doesn't show that submarines really swim; nor does it fail to establish the fact". Creating a computer that can produce millions of grammatical utterances does little to show that we understand language systems. Now, if a computer could - like humans - learn to produce infinite, novel, contextual, and meaningful grammatical utterances, that's a different story. But that story will take a lot more than statistical learning to write.

andrewcooke 15y ago
just because you only understand one side of an argument doesn't mean everyone else is an idiot. in some sense this is the same argument as searle's chinese room. that's sufficiently well known and debated that it's fair to say that neither side can be dismissed as simply "missing the point".
- brockf 15y ago
  Throughout history, there have been many "well known and debated" arguments that have proven to be idiotic. See: the shape of the Earth. This isn't one of them, though. And I didn't mean to imply that anyone was an idiot. Instead of critiquing how I said something, or that I said something at all, can you tell me what I am missing? I'm obviously missing something - Norvig is no idiot.
  - andrewcooke 15y ago
    the problem is whether or not there is any way to "ground" meaning. for physics, the "unreasonable effectiveness of mathematics" might suggest that there are simple "meanings" that underly physical "laws". but there's nothing to say that the same is true for intelligence or language. maybe the brain is nothing more than a particularly flexible "neural net", which statistical methods are modelling quite well. in that case, "intelligence" is not qualitatively different from "a good simulation of intelligence". the same problem occurs in free will - does it "really" exist? if we're just (mechanical, predictable, although highly complex) machines then it is difficult to imagine how it can. yet the intuition is that there is clearly some meaning to the idea of a "free agent". these are hard questions. people don't know the answers. instead we look to what daniel dennett calls "intuition pumps" (see his book "elbow room" on the free will problem) - simple parallels that "feel right". from those, we use intuition to argue in one direction or another. but the problem with that approach is that it depends on what you choose as a "hint". some advance is being made through experiment. imaging of neural activity in the brain, for example, or the recent discovery that people who believe they have free will behave differently to those that don't. searle's chinese room argument - http://plato.stanford.edu/entries/chinese-room/ http://plato.stanford.edu/entries/chinese-room/ unreasonable effectiveness of mathematics - http://www.jstor.org/pss/2321982 http://www.jstor.org/pss/2321982 http://plato.stanford.edu/entries/mathematics-explanation/ http://plato.stanford.edu/entries/mathematics-explanation/ i can't find the free will behaviour result that was in the news about a week ago. but that's perhaps less related to this anyway. and i doubt that everyone who thought the earth was flat was idiotic, frankly. just because something is obvious now doesn't mean it was a stupid question, or easy to answer, when first raised.

sethg 15y ago

This essay made me think: Lojban (http://www.lojban.org/tiki/la+lojban.+mo http://www.lojban.org/tiki/la+lojban.+mo), among constructed languages, is the categorial language par excellence. Every word has a well-defined range of meaning; the grammar can be parsed by the same kinds of parsers used for programming languages; potential sources of ambiguity, like plural references, associativity of modifiers, and negation, have been rigorously (or tediously, depending on how you roll) nailed down. Can there be such a thing as a conlang that demonstrates the ideal statistical grammar and semantics? (“All the words in this list are 60% likely to be used as nouns and 40% likely to be used as verbs....” But in the absence of a pre-existing linguistic community, how could you get students of the language to use them in the right proportions?)

borism 15y ago

And while it may seem crass and anti-intellectual to consider a financial measure of success Why are other metrics Norvig provides like articles published or prevalence in practical applications are considered more intellectual? And besides, I don't think "accurately modeling the world" is the end of it. Classical Newtonian mechanics correctly describe 99% of our activities in the real world and were considered pinnacle of scientific achievement for several centuries. Yet we know today that they're just a subset of General relativity and Quantum mechanics.

Jun8 15y ago

This is not a new debate. Within Linguistics there has been a continuous push against statistical NLP models. Read the introduction of Manning's book, even he seems to be defensive about NLP. Chomsky is a colossus, his achievements are well-known. However, at one point in many disciplines it comes to pass that the pioneers who pave the way in time become the very impediment to new ideas. His emphasis on Semantics have warped the minds of many generations of researchers (and some other ideas on universal grammar, too). I experienced this first hand, my advisor, Prof. Raskin, a great researcher on semantics, nevertheless thought that statistical approaches were not the way to go. Sadly, in many Linguistics departments people are just not equipped with the statistical tools necessary to have a basic understand of what's being done in the NLP field. So NLP is generally taught under CS, EE, or CompE.

adavies42 15y ago
i saw someone once compare chomsky to freud, as a foundational figure whose discipline can't/couldn't progress during his lifetime.
- ordinary 15y ago
  Einstein would be another example.
foldr 15y ago
>His emphasis on Semantics What are you talking about? Chomsky has always been highly critical of formal semantics.

noahlt 15y ago

Strangely appropriate is today's XKCD: http://xkcd.com/904/ http://xkcd.com/904/

kenjackson 15y ago
Hmm... I never thought of it that way. That sports are a weighted random number generator,but the various weights are unknown. And the commentators are discussing theories as to what the weights are, and how derived. (Although the cartoon seems to be saying the narratives are just about the numbers generated, which is more cynical, and frankly less interesting).
- yourcelf 15y ago
  Actually, Larry Birnbaum over at Northwestern is doing exactly that: http://infolab.northwestern.edu/projects/stats-monkey/ http://infolab.northwestern.edu/projects/stats-monkey/ They take the coded sports results, and automatically generate narratives using statistical speech models. They have a startup that is doing it too, don't recall the name of it.... EDIT: I believe this is it: http://narrativescience.com/ http://narrativescience.com/

T_S_ 15y ago

The handshake example was illuminating. Three "equivalent" theories: Theory A: Closed form formula function. Theory B: "Algorithm". Still a function. Theory C: Memoized function (constant time!) According to the article "nobody" likes C, especially the article's Chomsky straw man. If one had a procedure to convert C to A, then this whole issue would become hairsplitting. Such a procedure would aim to convert a memoized function back into a form that uses more symbols from a mathematical language. A good criteria of success would be the description length of the resulting procedure in the preferred language. One reason this could be useful to science is that once you identify a value that is useful in many theories it becomes part of the language. Making it available to the next problem may speed up the search for a "good" description of the next phenomenon. Identical procedures that appeared in various algorithms might acquire a special name. One such value might be called "pi", another "foldr" and so on. Of course there may be many good descriptions, just as there are many languages. Also, the example could be extended to statistical modeling situations by adding room for error terms in the suitability criteria. So if, you have a general procedure to convert a table into a definition you can make money and science at the same time!

niels_olson 15y ago

This whole theory vs observation argument exists at the very pinnacle of human thought, expressed in the Copenhagen interpretation. If you want to contribute to the human understanding of this, you'll have to beat Bohr and the uncertainty principle.

Create 15y ago
Fourier was there first.
- niels_olson 15y ago
  My claim wasn't first, it was top. You up-end the Copenhagen interpretation, show the universe really is deterministic, and every other argument on this subject, in every discipline, collapses. As it is, the arguments are almost certainly failed, but it's not quite a cinch. Because probability admits determinism as a special case. One of the deep points of Norvig's essay.

double-z 15y ago

The commentary has nothing to do with what Chomsky proposed. The author defines success as "being successful at accomplishing a task". That has nothing to do with science. Full stop.

macmac 15y ago

Are Norvig's comments on the "I before E except before C." really vaild? Why would one use a corpus for analysis of the rule, and not a dictionary? It appears to me that "CIE" (P(CIE) = 0.0014) is more common than "CEI" (P(CEI) = 0.0005) because the words that contain the "exception" "CIE" are used more frequently in the corpus than the words that follow the rule "CEI". Once you know the limited number of exceptions (in the dictionary sense) the rule appears to preserve its relevance.

jimbokun 15y ago
I suppose a the most useful corpus for this rule would be spelling tests.

stcredzero 15y ago

My conclusion is that 100% of these articles are more about "accurately modeling the world" then they are about "providing insight," although they all have some theoretical insight component as well. Before you can figure out why, you have to make sure you can accurately characterize the what. So there's a lot of science that is focused on coming up with a descriptive tool like an adhoc curve, before the underlying principles are discovered. I think Chomsky is afraid that statistical models will cause people to stop looking for the underlying principles.

davidmathers 15y ago

Chomsky called the Watson computer that won Jeopardy "a bigger bulldozer." He goes into more detail about his AI opinions here: http://www.framingbusiness.net/archives/1366 http://www.framingbusiness.net/archives/1366

galactus 15y ago

It is interesting than on a completely different debate, chomsky takes norvig's position (he is accused of not looking for a "theory" and "whys" and he replies that it is pragmatic results that matter): http://mindfulpleasures.blogspot.com/2011/01/noam-chomsky-on-derrida-foucault-lacan.html http://mindfulpleasures.blogspot.com/2011/01/noam-chomsky-on...

_grrr 15y ago

I've been monitoring the page this post points to with a bookmarking tool we've just released in beta. Here are the latest set of changes: http://app.bookmarkerpro.com/changes?fmt=html&id=2573 http://app.bookmarkerpro.com/changes?fmt=html&id=2573 Quite a few revisions since first posted to HN!

_grrr 15y ago
More revisions... http://tinyurl.com/3sabdc9 http://tinyurl.com/3sabdc9