13 ms·
On Chomsky and the Two Cultures of Statistical Learning (2011)
- dang 10y agoDiscussed at the time: https://news.ycombinator.com/item?id=2591154 https://news.ycombinator.com/item?id=2591154.
- atmosx 10y agoIn this video[1] Varoufakis takes on modern economics and their models as a way of understanding the real world and predict what's going to happen. On a higher level, I believe that he speaks for most (if not all) social sciences and what happens when they cross mathematical models who blindly try to understand and predict the real world through a flawed, limited mathematical model. [1] https://youtu.be/L5AUAIzciLE?t=1355 https://youtu.be/L5AUAIzciLE?t=1355
- pdkl95 10y agoMark Blyth says basically the same thing. https://www.youtube.com/watch?v=hmWbkPezgtU https://www.youtube.com/watch?v=hmWbkPezgtU
- ntoshev 10y agoThere is a problem with the simple economic models that don't claim to predict real world behavior because they capture only one aspect of reality (everything else being equal...). They are not really scientific because they are not falsifiable. Math works and it seems to reflect reality, but we have no epistemic mechanism to verify this. If we can do this eventually, then perhaps we can make our way to models that capture a larger fraction of the truth and can be tested against reality.
- x5n1 10y agoIf you have big data about all transactions occurring in real-time, along with socioeconomic factors on all the actors, it is not difficult to verify models and see how well the relationship between their dependent and independent variables holds up in the real world.
- mjohn 10y agoThat is a big if. Nobody has data for all transactions occurring in real time and nobody knows all the socioeconomic factors for all the actors. Even if you did have all of that data, you would still be missing a lot of data. For example, an important factor in economics is asymmetric information. An actor's decision may be optimal given the information available to them at the time even if the decision is suboptimal given perfect information. So for an even more accurate model, you would also need to have data on what each actor knows at any given point in time. Furthermore, an actor probably does not have the ability to calculate the optimal outcome given the information available and there are innumerable subjective factors in economic decision making. Big data will help you find some kind estimate, but it will not suddenly make economic modelling simple.
- mjohn 10y agoEconomic models that don't claim to predict real world behaviour are like thought experiments (with more maths). Their purpose is to help people understand that small isolated part of the whole, because a model that includes all the complexities of the real world would be incomprehensible. Nonetheless, many of these models are still falsifiable. For example, the textbook model of the benefits of trade due to comparative advantage only contains two countries and two goods. Although extremely simplistic, the theory does have falsifiable predictions and there is good empirical evidence supporting the broad theory. But this simple model will not be able to accurately predict how much a particular country will benefit from free trade.
- vezzy-fnord 10y agoCriticisms of mathematical modeling in economics were most definitely given by Mises and Hayek. Historian of economic thought Mark Blaug was also very critical of how neoclassical production theory (particularly Edgeworth's formulation) distorted the meaning of "competition" from a dynamic process to a quantity. Ironically, it is the same far-left heterodox demagogues like Varoufakis (though he's hardly the worst) who nevertheless use it in this sense to argue for mercantilist programs. Neoclassical when it suits them, heterodox when it doesn't. Anyone who uses the term "neoliberalism" is a charlatan. It is an absurd conspiracy theory that has since blown into an amorphous meme out of Marxist historiographers who struggle with the explanatory power (or lack thereof) of their framework. Mathematical models do have their uses in formalizing assumptions and analyzing dependencies, so they're not all bad, even if the Cowles Commission did go overboard. (Also let's not forget that the modern methodology of economics came as a result of the Keynesian research program, particularly since Hicks (1937)'s introduction of IS-LM, first modeled by Keynes himself in 1933 as four simultaneous equations.)
- AimHere 10y agoWhat makes the use of the term 'neoliberalism' to be a conspiracy theory? I thought it was a blanket term for the sorts of laissez-faire capitalist ideologies that sprang up since the 1980s, and whose proponents did have a big thing for namechecking Adam Smith and other classical liberal economists of the 19th century. You're surely not going to deny that there has been some sort of ideological drive in favour of free market reforms, privatization and "free trade" agreements of the WTO/TTIP ilk, are you? I don't think people who use the term think it refers specifically to shadowy cabal of free marketeers conspiring with each other over some hidden agenda. It's just a term for what a bunch of people, many of whom are in positions of power, happen to openly think and do.
- pessimizer 10y ago"Neoliberalism" is just an Americanism for economic liberalism, because the word liberalism is overloaded in the U.S., and is an autoantonym. That part of vezzy-fnord's comment is just a reflexive libertarian expulsion.
- conjectures 10y agoVaroufakis' philosophy is contra Chomsky I believe. When I've heard him speak, he attacks overly simplistic mathematical models which badly fit complex real world behaviour. If Chomsky's generative grammar provided a better model of language, Google translate would be running of that rather than statistical models. Chomsky's ideas don't explain the butterflies. In the same way Varoufakis targets theory led economics which don't explain recent events. Stats led models take the opposite approach, explaining the butterflies very well. The trade off is decreased ability to act in novel situations.
- asdasd23232sd 10y agoI would argue that this entire debate is about what it means to "explain".
- kposkdf 10y ago> If Chomsky's generative grammar provided a better model of language, Google translate would be running of that rather than statistical models. You realize google translate is utter crap right?
- YeGoblynQueenne 10y ago>> But isn't that the nature of scientific hypotheses? They are tenuous, only valid until you come across some data that tells you "aha, there's another piece to this rule". Chomskian grammars (CFGs) are used widely in compilers and similar tools to model computer languages and even for limited subsets of natural language they don't do half bad. The problem with phrase structure grammars is that they're very costly to develop and maintain and so far there's never been one such grammar that can model the whole of a natural language. If there was a way to learn CFGs with good coverage from text they'd be in much wider use, but unfortunately grammar induction is hard. Also, Google has its own political reasons not to want to use grammars. They champion neural networks and statistical AI. It's their schtick, innit.
- marmaduke 10y agoSame spirit as his critique of BF Skinner's theory of verbal behavior oh so many years ago.
- rooundio 10y agotl;dr "Classical" (natural) scientist: "X is only understood once I found first principles that x can be reduced to." "Modern" scientist: "X might be so complicated that there are no first principles that x can be reduced to. Rather, finding a neural network that can do x is the best I can do, and it explains why x can be done."
- mcguire 10y agoSee also * Statistical mechanics: https://en.wikipedia.org/wiki/Statistical_mechanics https://en.wikipedia.org/wiki/Statistical_mechanics
- foldr 10y agoI think you should delete the "first" in both cases. Chomsky merely wants a principled model. He's not making any pretense that we're anywhere close to understanding language at a fundamental level. Norvig isn't really interested in principled models at all.
- gcb0 10y agoit's important to know that chomsky used the same ideas in the 70s. he defended the biologicism (not sure about english translation of the term) which is basicaly machine learning in sociology. he fell on his face and is probably now in a very good position to throw this criticism.
- atdt 10y agoI'm not sure what you're talking about. Could you provide some references? Chomsky was extremely critical of "the Bell Curve", if that's what you're referring to. (http://newlearningonline.com/new-learning/chapter-6/chomsky-on-iq-and-inequality http://newlearningonline.com/new-learning/chapter-6/chomsky-...)
- gcb0 10y agoTry to find any of his papers from before 84. he was not a b.f. skinner fanboy, but pretty close, as was almost everyone on the field.
- faizshah 10y agoMore info on Chomsky's argument here: http://www.theatlantic.com/technology/archive/2012/11/noam-chomsky-on-where-artificial-intelligence-went-wrong/261637/ http://www.theatlantic.com/technology/archive/2012/11/noam-c...
- ntoshev 10y agoI wish there was a follow-up to this, Peter Norvig has hinted a few times he is going to write one.
- harry8 10y agoIt surprised me just how much of a piss-taking loon Norvig has turned out to be. How much fun do you think he had inventing the comparison of Chomsky to O'Reilly? Frickin' hilarious Pete, almost up there with the powerpoint presentation.
- thanatropism 10y agoIt seems to me that overarching theories of a Platonic bent miss the embodied-ness, Dasein-ness of human activity. Nevermind the debates about the ultimate nature of the human cognitive process; fact of the matter is that as observed, it's always-already wrapped in emotional-social thinking. Enough that there's reason to question the subject-object split altogether. Now, maybe Chomsky is a kind of extreme social-cognitivist and his abstract generative trees apply to societies as learning and meaning-producing wholes. But on the face of facts, rather than metaphysical speculation as to the nature of personality, intentionality and individuality, it would seem to me that the statistical/machine learning approach already faces language as it happens: as embodied in media, social context and so on. In other words: I fail to see much value in an abstract account of "pure language" as dissociated from the real communicative process as it happens right now as you read me. Sure, "insights" -- but it remains to be shown that "pure linguistics" is a worthwhile endeavor on the level of "pure quantum mechanics" as formal model.
- Ologn 10y ago> I fail to see much value in an abstract account of "pure language" as dissociated from the real communicative process But this assumption makes assumptions of purity as well. The communicative process may just be a byproduct of a mental process which has little to do with communication. A mutation happens tens of thousands of years ago (say 50,000 years ago), a change happens in the Broca (and/or Wernecke) area of the brain, and suddenly a new mental process kicks off. This mental process can be modeled as a state machine, and has the abilities and limitations of a state machine. It also has known limitations of output, which Chomsky has talked about. You're assuming the mutations which gave rise to the brain changes which created an internal language generator and parser have only one purpose - communication. But that's an assumption on your part. The ability to communicate may be just one byproduct of those changes which made things like communication possible.
- thanatropism 10y agoThat's an interesting point. Jacques Lacan insists on some ideas related to that. I've never been too fond on psychoanalysis either, but I've been known to be wrong often.
- fauigerzigerk 10y agoLet's consider the supreme court handshake problem, but say there is an unwritten social law that forces judges to only ever initiate a handshake with judges less senior than themselves. If the seniority of two particular judges happened to be exactly equal, they would not shake hands at all. Let's assume (I don't know if it is actually true or not) that in the history of the supreme court, there have never been two judges of the exact same seniority. In that case, a model learned from handshake data would not include the slightest hint of this unwritten social law. I think what Chomsky is saying is that if we do not understand the generative principle behind any data, we cannot possibly know what circumstance might completely invalidate our model. There may not be a way to smooth this out. Language understanding, contrary to things like speech recognition, does not lend itself very well to smoothing.
- netcan 10y agoIf this is unwritten (I take this to mean a piece of culture people practice without ever formalizing), how would the judges deal with the novel scenario? That rule will not have existed until it starts being practiced.
- fauigerzigerk 10y agoThe judges would simply apply the existing rule as it exists in their mind, but there is no trace of that state of mind in the handshake dataset. Yes the machine could guess. It could even guess correctly. But I think what Chomsky means is that we have no valid scientific reason to believe that it would.
- netcan 10y agoIf the machine is faced with this novel scenario before the judges, guesses, then that becomes the cultural norm.... does the machine win?
- ilitirit 10y ago> If the seniority of two particular judges happened to be exactly equal, they would not shake hands at all. Somewhat similar to Buridan's Ass. ... an ass that is equally hungry and thirsty is placed precisely midway between a stack of hay and a pail of water. Since the paradox assumes the ass will always go to whichever is closer, it will die of both hunger and thirst since it cannot make any rational decision to choose one over the other. https://en.wikipedia.org/wiki/Buridan%27s_ass https://en.wikipedia.org/wiki/Buridan%27s_ass
- mrow84 10y agoI think that Norvig hits the nail on the head near the beginning of his piece: "I believe that Chomsky has no objection to this kind of statistical model [the Newtonian model of gravitational attraction]. Rather, he seems to reserve his criticism for statistical models like Shannon's that have quadrillions of parameters, not just one or two." This is no more than an objection to problems of fitting your chosen model to data. If you only have a small number of free parameters, then you can fit your model with a reasonable amount of data. If you have a large number of parameters then you have to introduce some extra assumptions, as Norvig (of course) acknowledges slightly earlier (described as "smoothing", in context): "For example, a decade before Chomsky, Claude Shannon proposed probabilistic models of communication based on Markov chains of words. If you have a vocabulary of 100,000 words and a second-order Markov model in which the probability of a word depends on the previous two words, then you need a quadrillion (10^15) probability values to specify the model. The only feasible way to learn these 10^15 values is to gather statistics from data and introduce some smoothing method for the many cases where there is no data." Thus, although both models are statistical, it is much easier to have confidence in Newton's law of gravitation than it is in a Markov model of some communication channel, because the data tell a clear picture. The imprecision of Newton's law in certain parts of the problem space (unobserved during his time) is a moot point - any such objections apply equally well to models with many parameters, and then you _still_ have to accept that you have made extra assumptions "outside" the scope of your model. If you can explore your entire problem space, then you can build a complete "model". If not, then having more parameters than data _requires_ additional assumptions. Chomsky's point stands.
- grandalf 10y agoVery well articulated. One small point to add: Chomksy's interest has been in identifying the abstract characteristics of the language center of the human brain, which, for various reasons, does not seem likely to work like a Markov model. Analogously, one could look at the inputs and outputs of the human heart and potentially imagine a variety of physical structures that would explain them, and some of those physical structures would be biologically real/plausible and others would not. Things like constraints on working memory, exposure to input, and cross-language studies have informed the constraints that Chomsky has proposed to determine what kind of model would best capture the essential quality of the brain system. link: https://www.amazon.com/Knowledge-Language-Nature-Origins-Convergence/dp/0275917614 https://www.amazon.com/Knowledge-Language-Nature-Origins-Con...
- dschiptsov 10y agoThe language as a phenomena is obviously neither purely functional nor purely statistical. Purity is an abstract nonsense, an abstract category of abstractions. It is obvious from serious psychological studies of the process of a language acquisition, that it is similar to training a neural network - there is some knowledge representation grows up in the brain, but the process of training/learning is possible due to having appropriate machinery in the brain. It seems, like we have more that two apriory notions - of time and space, we, perhaps, have, apriory notions of a thing (noun), process (verb) and attribute (adjective) and even predicate at very least, as reflections of our perceptions of physical universe around us with sensory input procession machinery we happen to evolve. It is a mutualy recursive process - we evolved our "inner representation" of reality constrained by senses, but nature selects, in some cases, those with more correct representations. How these apriory notions maps to sounds - details of phonology and morphology is rather irrelevant - we evolved machinery for that. This is why, there is no fundamental, principal differences between human languages. The difference in in a degree, not in a kind. It seems also that we learn not the rules (schools are very recent innovations), but "weights" by being exported to the medium of a local spoken language. Children do it on their own, at least in remote areas, like among nomads of Himalaya, no worse than Americans. This, by the way, is prof that we have everything we need to be Buddha or Einstein. How exactly training occurs is absolutely unknown but it has nothing to do with probabilities. Nature knows nothing about probabilities, but it obviously "knows" rates - how often something happen. Animals "know" how often something happen. Probabilities is an invention of the mind, which leads to so many errors in cases where not all possible outcomes and its caused are know, which is almost always the case. Nature could not rely on such faulty tool. So, like every naturally complex system, it has both "procedures" and "weighted" data. Language capacity is hardwired, but grammar "grows" according to exposure. To speak about hows, and especially how-exactlys in terms of either pure procedures or pure statistics is misleading. It is both. And Mr.Chimsky is right - mere data, leave alone probabilistic models, describe nothing about principles behind what is going on. They does not even describe what's going on correctly, only some approximation to an overview of something unknown being partially observed. The more or less correct model, as a philosophy, must be grounded in reality, especially in that part of it which we call the mind. It has been pointed out, that mind itself is possible because of hardwired apriory notions (grounded in physical universe) of succession and distance, so models should be augmented with these notions too. Pure statistics is nothing.
- cschmidt 10y agoIf two cultures isn't enough for you, there was an interesting blog post from a year ago called "The Three Cultures of Machine Learning": http://cs.jhu.edu/~jason/tutorials/ml-simplex.html http://cs.jhu.edu/~jason/tutorials/ml-simplex.html
- indubitably 10y agoThis is totally off-topic, but man, Norvig writes some miserable HTML.
- pierrebai 10y agoOne can try to divine what Chomsky really thinks, believes and means, but one thing remain that always annoyed is is repated absolute stances and the way he express is viewpoint as being the only right one. Often expressed with disdain or dismissal of the opposition. You can disagree with someone, but doing it elegantly is of higher value to me.
- wfo 10y agoThis is generally the philosophical tradition: you present ideas, wholly and completely, as an explanation or a solution to a problem. You do not pre-suppose disagreements for your opponents (this will almost certainly end up being a straw man, if you let them make their own arguments they will make them more favorably than you could. It is a sign of respect). You express your idea as a firm, unyielding truth and wait until it is appropriately refuted, then reassess.
- mcguire 10y agoAnd when it is refuted, you assert something akin to: "Linguistic theory is mentalistic, since it is concerned with discovering a mental reality underlying actual behavior. Observed use of language ... may provide evidence ... but surely cannot constitute the subject-matter of linguistics, if this is to be a serious discipline."
- jdimov10 10y agoThere is no value in that. If you have a unique perspecitve, you MUST be outspoken and opinionated. Being quirky also helps. Otherwise, no one gets to hear about your unique perspecitve.
- PavlovsCat 10y agoIf you don't consider your viewpoint the most correct of all viewpoints known to you, why would you even have it? As for disdain, while I agree he can be overly dry sometimes, seeing the kind of stuff he often has to put up with, around political subjects anyway, I'd say he shows a lot of patience, too. If more people would do their part most of the more unpleasant subjects he debates people on wouldn't even be an issue. He wouldn't be able to use harsh words for, say, war criminals and their apologists, if those didn't exist in the first place. It's not actually his or anyone's job to try and help clean up the mess others are making, they do that on the side because it's required to look in the mirror. https://www.youtube.com/watch?v=g7bjZTmk0uU https://www.youtube.com/watch?v=g7bjZTmk0uU ^ This is what and who matters. Without a (decent) world, all the rest ain't happening anyway. He's the kind of person I'd imagine to be rather friendly to, say, a cleaning lady, and that to me are the more significant bits of the number that makes up grace. At least from what I see, he's sometimes a dick to people who are used to be lauded (and well paid), he's a champion of people who are used to get shit on. Furthermore: > "Courage is indispensible because in politics not life but the world is at stake." -- Hannah Arendt, "Between Past and Future" We're now at the point where we're not just worried about dying, or getting jailed, or losing a friend or two; but about merely being offended. Crooks and madmen aren't content anymore with merely getting away with it, now they want respect, too. I'm not even sure how polemic or exaggerated that is. Criminals are tired of laundering money so to speak, they just want to whitewash the crimes themselves and do them in the open. You can respectfully disagree in a way that leaves room for you not actually being sure of what you're saying, and that's it. Also, saying "this house is on fire" is not an "absolute stance" on fires or building safety, and if the house is on fire, and as long as no serious efforts are made to change that, it's simply principled to repeat this over and over, instead of coming up with something "new" for the sake of it. Hannah Arendt again: > The ceaseless, senseless demand for original scholarship in a number of fields, where only erudition is now possible, has led either to sheer irrelevancy, the famous knowing of more and more about less and less, or to the development of a pseudo-scholarship which actually destroys its object. I think this could also be applied to discussion of politics. Every time a government or the NSA does something, a bunch of people say "why does this surprise anyone?", as if something that sucked on day 1 would somehow be less alarming on day 50, with the same or even increased levels of suckage. Same for Chomsky criticizing his own, which is still valid. Why does it have to be new, or elegant, or otherwise pleasing? If a society shits its pants, that's already not something pleasant to point out, but the more years pass, the longer it keeps sitting in and adding to it, the less pleasant it becomes. It's not anyone's job to add sugar to the medicine, or give it with a big smile, or anything. I think it's awesome if they do, but still perfectly fine if they don't. Insofar his or anyone's arguments have merit, you can always repackage it without any vitriol or snark, and they stay intact. That is what really matters, and that kind of elegance is also in the ear of the listener.
- 3pt14159 10y agoChomsky is wrong that statistical models of language provide no insight. They do. Chomsky is right that language has meaning and that many modern statistical techniques essentially ignore this. My take on the issue is that you can't separate linguistic command from true intelligence / cognition. There's a long tail of tricks that us intelligent people can use, but fundamentally they'll only be tricks. And if we truly get something resembling a perfect linguistic-aware AI by this long tail of tricks then we've probably accidentally created real cognition. Maybe after typing this all out I finally understand what Turing meant.
- foobarqux 10y agoWhat fundamental understanding have statistical models produced?
- mcguire 10y agoSee statistical mechanics: https://en.wikipedia.org/wiki/Statistical_mechanics https://en.wikipedia.org/wiki/Statistical_mechanics [Edit] No really, take the ideal gas law. To get a real understanding of a given situation, you would have to know the position and momentum of all of the molecules of the gas. But the ideal gas law says, no, you don't need all of that to get a very good approximation; all you need to know to get the pressure of a gas is the temperature, volume, and amount of gas.
- foobarqux 10y agoObviously not the type of statistical model I was referring to.
- mcguire 10y agoObviously? Physics went through this same argument with thermodynamics and statistical mechanics. Predictive, statistical models pretty much won. And then along came quantum mechanics---there's nothing descriptive there at all.
- UhUhUhUh 10y agoAgain, this boils down to the rationalist vs. empiricist stance. Hidden variable vs. probabilistic approximation and so forth. The partisan error is to consider these two stances as mutually exclusive. They are not, as illustrated by decoherence for example. On the other hand, it is, I believe, undeniable that the rationalist endeavor is much more complex than the empiricist one and therefore also less linear. Less predictable and, please, let's not forget, less financially profitable. Chomsky's position has always been to advocate for the rationalist stance in a world conveniently inebriated with its empiricist successes to the point of turning this one aspect of thinking into a belief system. It is a waste of energy to argue for a monopoly of the yin over the yang or the other way around. They are complementary but not mutually exclusive. Privileging one over the other will result in an increase of ideological/mystical bias and essentially miss the whole point: it is their interaction, the transition from and to one another, that holds the key to a global understanding.
- zump 10y agoWhat does this mean for the YC chatbot startups?!
- mcguire 10y agoChomsky's aversion to statistical techniques is much deeper than most of this discussion focuses on. Here's an enlightening quote from Chomsky: "Linguistic theory is concerned primarily with an ideal speaker-listener, in a completely homogeneous speech community, who know its (the speech community's) language perfectly and is unaffected by such grammatically irrelevant conditions as memory limitations, distractions, shifts of attention and interest, and errors (random or characteristic) in applying his knowledge of this language in actual performance. (Chomsky, 1965, p. 3)" Chomsky is uninterested in linguistic data of the kind used to build statistical language models; those are "linguistic performances" and he is only looking at "linguistic competence", the ability of an ideal speaker to "produce and understand an infinite number of sentences in their language, and to distinguish grammatical sentences from ungrammatical sentences."[1] Now, I'm personally happy to criticise statistical techniques for their lack of explanatory power. But I'm not willing to go further and say that data is irrelevant. Chomsky is. [1] https://en.wikipedia.org/wiki/Linguistic_competence https://en.wikipedia.org/wiki/Linguistic_competence
- foldr 10y agoChomsky is not saying that "data is irrelevant". Aspects contains a detailed discussion of lots of linguistic data points, as you'd know if you weren't just cherry-picking quotations.
- fouc 10y agoThis seems a little ironic, I feel there's some incredibly useful things in AI/ML that rarely get applied to real world problems, instead academics just spend all their time trying to unravel the black box behavior and come up with some model for it.