13 ms·
Chomsky and the Two Cultures of Statistical Learning (2011)
- templar_snow 10mo ago[flagged]
- Epa095 10mo agoAnd? The article by Peter Norvig is still interesting.
- edm0nd 10mo ago[flagged]
- avmich 10mo agoDoes it matter?
- derriz 10mo agoA lot of Chomsky’s appeal I believe is due to his politics as his universal grammar theories turned out to be an academic dead end. But his politics centers around the moral failings of the West so I think yes, if he was involved in the sexual exploitation of trafficked children, then this would devalue his criticism of the morality of the Western political system.
- breppp 10mo agoHis criticism of the Western political system was always way too simplicist and why it has immense appeal to college students. Essentially it can be summed as any Western action must be rationalized as evil, and any anti-west action is therefore good. This is also in line with Christian dualism so the cultural building blocks are already in place. Then you get Khmer Rouge, Putin, Hezbollah, Iran apologetism or downright support
- derriz 10mo agoI am not a fan of Chomsky - the opposite in fact. I was deliberately avoiding judging his actual arguments - to make the point that his own morality undermines his lecturing others on their moral failings.
- throwaway_dang 10mo agoI doubt you can find any essay or such where he said anti-Western action was good on the sole grounds that it was anti-Western. It's difficult to summarise so many years of writing in a few sentences but from my own reading, he pointed out a) many things done by the US lead to death or destruction b) many of these things are justified in the name of good that doesn't stand up to scrutiny c) the US government is often hypocritical d) US citizens are heavily propagandized both for foreign policy and domestic policy e) as a US citizen, it his duty to try and oppose these actions and since he's not a citizen of Iran, he isn't in a position to do anything about Iran f) a) through d) explain why he is often seen as an apologist, to use your word, for Iran; he tries to explain, from his point of view, why Iran etc. do the things they do g) a strong support of freedom of speech and opposition to censorship, including what he regards as private censorship as opposed to merely government censorship.
- breppp 10mo agoThat doesn't explain why he visited Hezbollah and showed overwhelming support, probably aware of the organization roots and past actions such as kidnapping journalists or killing politicians or its self professed goal of creating a theocracy in Lebanon. He of course has very complex rationalizing but essentially he assumes the opposite of mainstream western opinion and then tries to build ideological structures upon that. That creates a very simplified version of reality wrapped in a nice intellectual wrapping
- throwaway_dang 10mo agoChomsky had been involved in linguistics and politics since the 60s, which is nearly six decades covering a multitude of events and issues. To simplify his work down to even a paragraph is an impossible task, let alone as you have done as simply saying "anti Western". For example, during the 2003 US invasion of Iraq, Germany and France were opposed to the invasion, leading to "Freedom Fries" to insult French opposition to the war. The British public was also opposed to the war, although the the Blair government went along with it anyway. Australia had a similar position - public opposition but government went along with it anyway. Canada official refused entry into the Iraq war. Chomsky was also opposed to the Iraq war. Does this mean that France, Germany, Canada and the British and Australian general public are "anti-Western"? Since Chomsky agreed with these countries, does that make him anti-Western or pro-Western? Does it make the US anti-Western since they proceeded with a war despite formal or popular opposition in many Western countries? I fear you have a certain definition of the "Western" that simply excludes Western opinions that don't fit your understanding. As to who Chomsky met him; well as part of this Epstein story, Chomsky met with former Israeli prime minister Ehud Barak. In your opinion, does this make him anti-Western? Indeed, prior to his stroke Chomsky explained that this kind of meeting is why Chomsky associated with Epstein - for the contacts. I suspect Chomsky is just generally interested in understanding an issues and not bothered by what it's seen as, seemingly to his detriment in this Epstein story.
- plastic-enjoyer 10mo ago> But his politics centers around the moral failings of the West so I think yes, if he was involved in the sexual exploitation of trafficked children, then this would devalue his criticism of the morality of the Western political system. Why would it devalue his criticism assuming he was right?
- derriz 10mo agoMoral arguments for me don’t stand alone like a mathematical proof or scientific findings which can be examined as some sort of platonic form. Morality arguments are social and contextual. That 2+2 is 4 won’t change and captures some sort of eternal truth while what is deemed moral is constantly changing over time and differs across different societies and social groupings. So morality arguments require and appeal to a particular shared sense of right and wrong. If Chomsky was guilty of sexually abusing children, then I do not share his moral foundation and so his appeals to morality arguments do not convince me.
- plastic-enjoyer 10mo agoDo you have an example where Chomsky might be right but you disagree with him because of his moral depravity?
- derriz 10mo agoWhy? There are some of Chomsky’s positions I’m sure I agree with and some I disagree with. What’s the relevance to my point?
- plastic-enjoyer 10mo agoIf it turns out that Chomsky was sexually abusing children would you start disagreeing with Chomskys positions you agreed previously?
- derriz 10mo ago
- retrac 10mo agoI won't try to defend Chomsky. (Not really a big fan even before this.) But if the mere mention of him is sus to you then I advise you to not study either linguistics or computer science because it's Chomsky normal forms and Chomsky hierarchies all the way down. There's even still people clinging to some iteration of the universal grammar despite the beating it has taken lately. He's also one of the most prominent political thinkers on the American hard left for the last half century. There's a joke going around for a while now that you either know Chomsky for his politics, or for his work in linguistics and discrete mathematics, and you are shocked to discover his other work. I guess we can extend that to a third category of fame, or infamy.
- cma 10mo agoThe merge operation in the later Chomsky modern linguistics program is similar in a lot of ways to transformer's softmax merging of representations to the next layer. There's also still a lot to his arguments that we are much more sample efficient. And it isn't like monkies only learn language at a gpt-2 level, bigger brains take us to gpt-8 or whatever. There's a step change where they don't really pick things up linguistically at all and we do. But with a lot more data than we ever get, LLMs seem to distill some of the broad mechanisms what may be our innate ability, though still seems to have a large learned component in us.
- eru 10mo agoNot sure that's relevant? People still discuss what Einstein did, and he's long dead. (I don't like Chomsky for other reasons, but having an obituary ain't no reason to disregard someone's thoughts.)
- poaching 10mo agoWho else in tech/AI did they whale?
- mmooss 10mo agoAre you implying Norvig is a victim or otherwise not responsible for their choices and actions?
- deleted 10mo ago[deleted]
- pmkary 10mo agoDude would talk about manufacturing consent, elitist circles, and what Israel is doing with poor Palestinians and then go aboard Israeli-spy, super elitist, consent manufacturing, sex trafficker, rapist, Epstein's private jet. What a total insult to everyone who ever read his things
- ra 10mo agoHopefully he'll have something to say on it
- throwaway_dang 10mo agoChomsky had a stroke a couple of years ago and isn't capable of speaking; the family is trying to maintain their privacy and so there isn't much public information about it but it came out that he can raise his arm when he sees something he dislikes and it doesn't look like much beyond that.
- emsign 10mo agoHe already said he had nomoral objections to deal with Epstein knowing about his first conviction for sex trafficking, because in Chomsky's view the man served his time and justice had been served. Yes, to Chomsky Epstein was an innocent man after serving a few months for sex trafficking and having sex with a dozen of minors. The socialist anarchist Chomsky had no ethical objections when he asked a convicted billionaire sex trafficker how to invest a few millions.
- userbinator 10mo agoAlong with a bunch of other, arguably far more famous people.
- throwaway_dang 10mo agoBut there is no indication or even accusation that he was involved in any sexual activity, let alone anything inappropriate. It's innuendo and guilt by association, mainly by his political opponents, both on the left and right, that are taking advantage of his inability to defend himself due to his stroke. I think many people are being _justly maligned_ by their association with Epstein, but in a way that distracts from the wider issue of what exactly does it mean when so many powerful and prominent people are found in compromising or potentially compromising situations and to what ends it served. It's US kompromat and the discussion is largely restricted to maligning people without discussing the significance of it. In terms of Chomsky himself, given his career spanned both linguistics and politics, an honest critique would either deal with their disagreements with Chomsky like how Norvig did in this essay, or how Hitchens did over the Afghan and Iraq wars rather than saying "he had dinner with Epstein" or "he had dinner with Bannon". In terms of the Epstein issue, the best criticism I can see is that his association with Epstein, Bannon etc. makes him a hypocrite although I don't find this personally convincing. Part of the problem for me here is that his present infirmities make it difficult for him to defend or explain himself and I find it poor form to kick the man when he's down, mainly by people who just didn't like that Chomsky didn't agree with them personally. Especially when he largely made a contribution to the debates even if one doesn't agree with him.
- deleted 10mo ago[deleted]
- intalentive 10mo agoThis essay is missing the words “cause” and “causal”. There is a difference between discovering causes and fitting curves. The search for causes guides the design of experiments, and with luck, the derivation of formulae that describe the causes. Norvig seems to be confusing the map (data, models) for the territory (causal reality).
- gsf_emergency_6 10mo agoA related* essay (2010) by a statistician on the goals of statistical modelling that I've been procrastinating on: https://www.stat.berkeley.edu/~aldous/157/Papers/shmueli.pdf https://www.stat.berkeley.edu/~aldous/157/Papers/shmueli.pdf To Explain Or To Predict? Nice quote We note that the practice in applied research of concluding that a model with a higher predictive validity is “truer,” is not a valid inference. This paper shows that a parsimonious but less true model can have a higher predictive validity than a truer but less parsimonious model. Hagerty+Srinivasan (1991) *like TFA it's a sorta review of Breiman
- 0928374082 10mo agois it more than a commentary on overfitting to the tune of "with enough epicycles you can make the elephant wiggle its trunk"?
- gsf_emergency_6 10mo agoIf you are referring to Hagerty+Srinivasan: They certainly didn't think that a better fit => "truer". They used the term "truer" to describe a model that more accurately captures the underlying causal structure or "true" relationship between variables in a population. As for the paper I linked, I still haven't read it closely enough to confirm that D-Machine's comment below is a good dismissal. I'm inclined to think it's more like "interpolating vs extrapolating"
- tripletao 10mo agoThis essay frequently uses the word "insight", and its primary topic is whether an empirically fitted statistical model can provide that (with Norvig arguing for yes, in my opinion convincingly). How does that differ from your concept of a "cause"?
- bo1024 10mo agoIs this essay from 2011?
- deleted 10mo ago[deleted]
- tripletao 10mo agoHere's Chomsky quoted in the article, from 1969: > But it must be recognized that the notion of "probability of a sentence" is an entirely useless one, under any known interpretation of this term. He was impressively early to the concept, but I think even those skeptical of the ultimate value of LLMs must agree that his position has aged terribly. That seems to have been a fundamental theoretical failing rather than the computational limits of the time, if he couldn't imagine any framework in which a novel sentence had probability other than zero. I guess that position hasn't aged worse than his judgment of the Khmer Rouge (or Hugo Chavez, or Epstein, or ...) though. There's a cult of personality around Chomsky that's in no way justified by any scientific, political, or other achievements that I can see.
- dleeftink 10mo ago> novel sentence The question then becomes on of actual novelty versus the learned joint probabilities of internalised sentences/phrases/etc. Generation or regurgitation? Is there a difference to begin with..?
- tripletao 10mo agoI'm not sure what you mean? As the length of a sequence increases (from word to n-gram to sentence to paragraph to ...), the probability that it actually ever appeared (in any corpus, whether that's a training set on disk, or every word ever spoken by any human even if not recorded, or anything else) quickly goes to exactly zero. That makes it computationally useless. If we define perplexity in the usual way in NLP, then that probability approaches zero as the length of the sequence increases, but it does so smoothly and never reaches exactly zero. This makes it useful for sequences of arbitrary length. This latter metric seems so obviously better that it seems ridiculous to me to reject all statistical approaches based on the former. That's with the benefit of hindsight for me; but enough of Chomsky's less famous contemporaries did judge correctly that I get that benefit, that LLMs exist, etc.
- dleeftink 10mo agoMy point is, that even in the new paradigm where probabilistic sequences do offer a sensible approximation of language, would novelty become an emergent feature of said system, or would such a system remain bound to the learned joint probabilities to generate sequences that appear novel, but are in fact (complex) recombinations of existing system states? And again the question being, whether there is a difference at all between the two? Novelty in the human sense is also often a process of chaining and combining existing tools and thought.
- barrenko 10mo agoIs this bayesian vs. frequentist?
- tgv 10mo agoIn one word: no. In more detail: Chomsky is/was not concerned with the models themselves, but rather with the distinction between statistical modelling in general, and "clean slate" models in particular on the one hand, and structural models discovered through human insight on the other. With "clean slate" I mean models that start with as little linguistically informed structure as possible. E.g., Norvig mentions hybrid models: these can start out as classical rule based models, whose probabilities are then learnt. A random neural network would be as clean as possible.
- barrenko 10mo agoThank you!
- deleted 10mo ago[deleted]
- ur-whale 10mo ago[flagged]
- codeulike 10mo ago(this is from 2017)
- atomicnature 10mo agoIt was available earlier. Here's the HN history: https://hn.algolia.com/?query=Chomsky%20and%20the%20Two%20Cultures%20of%20Statistical%20Learning&type=story&dateRange=all&sort=byDate&storyText=false&prefix&page=0 https://hn.algolia.com/?query=Chomsky%20and%20the%20Two%20Cu... The oldest submission is from 15 y.o ago - that is 2010. I resubmitted it - thinking - with the success of LLMs - felt this was worth a revisit from "how real-world scientific progress works" point of view.
- cubefox 10mo agoNo it is from 2011. The text mentions an event in 2011, so it couldn't have been written earlier, and the first HN submission [1] was in 2011, so it also wasn't written later. The title should say (2011), otherwise the whole piece is confusing. 1: https://news.ycombinator.com/item?id=2591154 https://news.ycombinator.com/item?id=2591154
- pmkary 10mo agoI have many books from Chomsky, and I want to throw them away because it disgusts me to have them. Then I think, why should I throw away things I spent so much on? It makes me more angry. So I have pilled them up somewhere to figure out what ti do with them and each time I walk past it I feel sad to ever passed by his work.
- f1shy 10mo agoI assume this comes from his views in politics and/or association with things like Epstein. I would say, independent of that, some works of him can be very valuable. Private life of persons and their work, are better put in totally different context, and not mixed.
- spwa4 10mo agoThe thing is, nothing that usually changes things applies to Chomsky. What he did was most certainly not a normal thing to do in his time. Like one might say about George Washington or even further back, like Clovis. By today's standards they were morally wrong, but not by the standards of their time and they advanced morals. They made things better. Chomsky is wrong by the standards of his time and is making things worse rather than better. It was very much the opposite of Chomsky's ideology as well. So it additionally means he's fake. BOTH on his morals and politics/activism, from both sides (ie. both helping a paedophile, and helping/entertaining a billionnaire). So it's (yet another) case of an important figure that supposedly stands for something, not just demonstrating he stands for nothing at all, but being a disgusting human being as well.
- cubefox 10mo ago(2011)
- deleted 10mo ago[deleted]
- deleted 10mo ago[deleted]
- scottoreily 10mo ago[dead]
- oldpersonintx2 10mo ago[dead]
- MoravecsParadox 10mo ago> derided researchers in machine learning who use purely statistical methods to produce behavior that mimics something in the world, but who don't try to understand the meaning of that behavior. It's crazy how wrong Chomsky was about machine learning. Maybe the real truth is that humans are stochastic parrots who have an underlying probability distribution - and because gradient descent is so good at reproducing probability distributions - LLMs are incredibly good at reproducing language.
- AuthAuth 10mo agoIs it crazy? Chomsky is wrong on so many of the topics he speaks about.
- cboyardee 10mo ago[dead]
- adamddev1 10mo ago> Chomsky has focused on the generative side of language The answers to "why" that Chomsky pushes so hard for are very valuable to adult language learners. There are basic syntactic rules to generating broadly correct language. Having these rules discovered and explained in the simplest possible form is irreplaceable by statistical models. Neural networks, much like native speakers can say "well this just sounds right," but adult learners need a mathematical theory of how and why they can generate sentences. Yes, this changes with time and circumstances, but the simple rules and theories are there if we put the effort in to look for them. There are many languages with a very small corpus of training data. The LLMs fail miserably at communicating with them or explaining things about their grammar, but if we look hard for the underlying theories Chomsky was looking for, we can make huge leaps and bounds in understanding how to use them.