5 ms·
As someone who tends to side with Chomsky in these debates, I think Norvig makes some interesting points. However, I would like to pick one of his criticisms to
by reliableturing 4y ago
As someone who tends to side with Chomsky in these debates, I think Norvig makes some interesting points. However, I would like to pick one of his criticisms to disagree on:
"In 1969 he [Chomsky] famously wrote:
But it must be recognized that the notion of "probability of a sentence" is an entirely useless one, under any known interpretation of this term.
His main argument being that, under any interpretation known to him, the probability of a novel sentence must be zero, and since novel sentences are in fact generated all the time, there is a contradiction. The resolution of this contradiction is of course that it is not necessary to assign a probability of zero to a novel sentence; in fact, with current probabilistic models it is well-known how to assign a non-zero probability to novel occurrences, so this criticism is invalid, but was very influential for decades."
I think Norvig wrongly interprets Chomsky's "probability of a sentence is useless" as "the probability of a novel sentence must be zero". I agree that we've shown that it's possible to assign probabilities to sentences in certain contexts, but that doesn't mean that it can fully desribe a language and knowledge. This seems to me yet another case of 'the truth is somewhere in the middle' and would be weary of the false dichotomy that is put forward here. Yes we can assign probabilities to sentences and they can be useful, but it's not the whole story either.
- KKKKkkkk1 4y agoI'm sorry, but what Chomsky wrote is pseudo-scientific mumbo jumbo, and Norvig went to the trouble of wading through that mumbo jumbo and refuting it. Thank you, Peter Norvig.
- dekhn 4y agowhat's really funny is that I worked in DNA sequence analysis at a time when the chomsky hierarchy was primal and literally all my work was applying "probability of a sequence" concepts (specifically, everything from regular grammars to stochastic context free grammars). It's a remarkably powerful paradigm that has proven, time and time again, to be useful to scientists and engineers, much more so than rule systems constructed by humans. The probability of a sentence is vector-valued, not a scalar, and the probability can be expanded to include all sorts of details which address nearly all of Chomsky's complaints.
- avgcorrection 4y agoChomsky’s theories of human language weren’t useful to your work on DNA? That is funny. Linguists everywhere in shambles.
- dekhn 4y agoYou misunderstand. The chomsky hierarchy was critical to my work on DNA. In fact my whole introduction to DNA sequence analysis came from applying probabilistic linguistics: see this wonderful paper by Searles: https://www.scribd.com/document/461974005/The-Linguistics-of-DNA https://www.scribd.com/document/461974005/The-Linguistics-of...
- avgcorrection 4y agoI still don’t understand. What’s funny about it and what’s DNA got to do with human language sentences (the original quote)? Chomsky’s grammars are used in compilers and compiler theory. Even though programming languages have got nothing to do with human languages. Certainly nothing to do with the “probability of a sentence” that he was talking about. The application of something like that doesn’t necessarily tell you anything about what Chomsky is talking about, namely human language.
- dekhn 4y agoFunny you mention that- after working with probabilistic grammars on DNA for a while, I asked my advisor if the same idea could be applied to compilers, IE, if you left out a semicolon, could you use large number of example programs to train a probabilistic compiler to look forward enough to recognize the missing semicolon and continue compiling? They looked at me like I was an idiot (well, I was) and then said very slowly.... "yes, I suppose you could do that... it would be very slow and you'd need a lot of examples and I'm not sure I'd want a nondeterministic parser". My entire point above is that Chomsky's contributions to language modelling are the very thing he's complaining about. But what he's really saying is "humans are special, language has a structure that is embedded in human minds, and no probabilistic model can recapitulate that or show any sign of self-awareness/consciousness/understanding". I do not think that humans are "special" or that "understanding" is what he thinks it is.
- originalcopying 4y agobut what about things with probability zero that nonetheless happen all the time? I'm not very clear on some technicalities around probability, but I remember a 3blue1brown video: something along the lines of the probability of randomly choosing an integers out of the real number line being a probability zero event; in spite of there being infinitely many integers to "randomly pick"
- time_to_smile 4y agoWhen was the last time you picked a random number from an actual real number line? Try sampling random floats in your programming language of choice and see how long it takes to get an integer (you will eventually get one). Then consider that floating point numbers represent only a finite subset of the (countably) infinite rational numbers. And then consider that the set of rational numbers is unimaginably smaller than the set of irrational numbers (which is the other part of the reals). The fact that integers have measure 0 in the set of real numbers only seems confusing if you're thinking of everyday operations rather than thinking about mathematical abstractions.
- deleted 4y ago[deleted]
- dyno12345 4y agoAndrey Markov would like a word
- deleted 4y ago[deleted]