These are fair criticisms but imo miss Chomsky's much bigger point. From Chomsky's essay, 2 important lines: "Whereas humans are limited in the kinds of exp

These are fair criticisms but imo miss Chomsky's much bigger point. From Chomsky's essay, 2 important lines: "Whereas humans are limited in the kinds of explanations we can rationally conjecture, machine learning systems can learn both that the earth is flat and that the earth is round. They trade merely in probabilities that change over time. For this reason, the predictions of machine learning systems will always be superficial and dubious." He's saying we trust humans because they can say things like "I'm pretty sure X because of explanation Y" and under the hood, we process the explanation, form our own probability of X and trust our own computation. But since LLMs cannot provide explanations for their beliefs, humans will never be able to rely on LLMs because the way we actually communicate is through explanations, not probabilities. Chomsky's conclusion: this is a good predictor, but not a human. Where Chomsky is actually wrong is he mixes up how good are you at prediction vs. how well can you convince a human of the prediction vs. how intelligent you are. We humans use a combination of accuracy + convincingness as a marker of intelligence. An AI that was just as accurate but 0 ability to convince a human because 0 ability to produce explanations could still be intelligent. This AI would seem like an alien to us, but an alien that could uncannily beat us at any challenge that requires an understanding of the natural world. In fact, we might never truly understand its internal explanations of the world, but still acknowledge it has them and is good at building new ones internally. Imagine being in a room with a foreign language speaker who beats you at chess. Clearly they have a mental model that works, even if they cannot explain it to you.