3 ms·

Once you go beyond Markov chains and into vector models (Transformers or even LSTMs), next —-token-- syllable prediction can capture grammar.

by stared 2y ago

foldr 2y ago
That some aspects of grammar can be captured by statistical analysis was never really in dispute. The OP is slightly confused about the hierarchy part. Chomsky never said that you couldn't discover by statistical analysis that languages have a hierarchical structure. He rather said that a baby, based on the data it has available, would not be able to determine statistically that certain rules of grammar are defined over this structure rather than over the linear sequence of words.
- stared 2y ago
  I mean, contrary to Chomsky, that certainly advanced statistical models are able to capture all grammar - at least, nit worse than ordinary humans. It feels that Chomsky’s understanding of ML stopped at the level of Markov chains.
  - foldr 2y ago
    Is there any work demonstrating this? For example, how do statistical models capture adjunct/argument asymmetries in extraction?
    - stared 2y ago
      Attention is one of the core parts of transformer architecture, so I would be surprised if they have any trouble understanding this asymmetry. Could you provide a testable hypothesis? I would be happy to test it on GPT4.
      - foldr 2y ago
        Sure, here are a couple of examples of ECP violations removing ambiguities. 1a. How often did you tell John that he should take out the trash? b. How often did you tell John why he should take out the trash? (1a) can either be a question about frequency of telling or frequency of trash disposal, whereas (1b) can only be a question about frequency of telling. I asked GPT-4 to explain how each sentence was ambiguous and it seemed to entirely miss the embedded readings (the ones about frequency of trash disposal) for both sentences, while finding some other ambiguities that were spurious (such as suggesting erroneously that (1b) could be a question about how many different reasons you gave John in a single instance). Similarly, (2a) has both a de re and a de dicto reading, whereas (2b) has only a de re reading: 2a. How many books did Bill say that Mary should read? b. How many books did Bill explain why Mary should read? That is, (2a) can be asked either in a scenario where Bill has said "read 10 books!" or in a scenario where Bill has said "read Book A, Book B and Book C!" without necessarily counting the books himself. (2b), on the other hand, only has the second kind of interpretation. I've had mixed results with GPT-4 in this case (depending on exact choices of vocabulary, etc.), but it certainly makes some mistakes. For example, it says that (2b) can mean "John explained the reason for a certain number of books that Mary bought". As the sibling comment points out, it would not show very much if GTP-4 did correctly determine these ambiguities as it has had access to much more data than a child. You would also need to show that the same statistical techniques would work when applied to a realistic dataset.
        stared 2y ago
        Thank you for providing these examples. I asked GPT4o, and it has no trouble with understanding 1a: https://chatgpt.com/share/daea469f-d823-45e2-9d6b-f6bea82a2623 https://chatgpt.com/share/daea469f-d823-45e2-9d6b-f6bea82a26... As a side note, my instinctive reading is on the telling frequency. Sure, one can make a garden path sentence, but (for my own ESL eyes and ears) it would be more straightforward to say, "Did you tell John how often he should take out the trash?" 2b does not feel right on its own (and I am not an AI). I can understand it, but it feels like reverse engineering rather than reading a normal sentence.
        foldr 2y ago
  - foobarqux 2y ago
    First, Chomsky (and I) were talking about language acquisition in children where there aren't billions of examples, so it's completely irrelevant if some other system can do something, the question is how do humans do it. Second, there isn't any evidence that LLMs have captured grammar rules in any meaningful sense, just as they can't do addition or any other recursive computation.
  - numpad0 2y ago
    Don't waste your time arguing Chomsky supporters, it's a cult. They keep wringing non-falsifiable theories out of lengthy hallucinations, but it all reveals itself to be just Trek physics later on. He's not just problematic in politics, he is in linguistics too, just the BS is harder to notice. I suppose his compiler theories were legit, and his pioneering spirit that lead to establishment of multiple fields of researches might be too, but leading theories he'd created are just as dubious as the first man made shed built on a discovered island would be. And the problem is not just that early speculations in a novel field are often wrong, it's that his supporters don't care. They'd regurgitate those scientific theories(tm) ad infinitum and waste resources for the whole humanity. So don't bother trying to fix them and making them motivated.

Once you go beyond Markov chains and into vector models (Transformers or even LSTMs), next —-token-- syllable prediction can capture grammar.

by stared 2y ago

Once you go beyond Markov chains and into vector models (Transformers or even LSTMs), next —-token-- syllable prediction can capture grammar.

foldr 2y ago

That some aspects of grammar can be captured by statistical analysis was never really in dispute. The OP is slightly confused about the hierarchy part. Chomsky never said that you couldn't discover by statistical analysis that languages have a hierarchical structure. He rather said that a baby, based on the data it has available, would not be able to determine statistically that certain rules of grammar are defined over this structure rather than over the linear sequence of words.

stared 2y ago
I mean, contrary to Chomsky, that certainly advanced statistical models are able to capture all grammar - at least, nit worse than ordinary humans. It feels that Chomsky’s understanding of ML stopped at the level of Markov chains.
- foldr 2y ago
  Is there any work demonstrating this? For example, how do statistical models capture adjunct/argument asymmetries in extraction?
  - stared 2y ago
    Attention is one of the core parts of transformer architecture, so I would be surprised if they have any trouble understanding this asymmetry. Could you provide a testable hypothesis? I would be happy to test it on GPT4.
    - foldr 2y ago
      Sure, here are a couple of examples of ECP violations removing ambiguities. 1a. How often did you tell John that he should take out the trash? b. How often did you tell John why he should take out the trash? (1a) can either be a question about frequency of telling or frequency of trash disposal, whereas (1b) can only be a question about frequency of telling. I asked GPT-4 to explain how each sentence was ambiguous and it seemed to entirely miss the embedded readings (the ones about frequency of trash disposal) for both sentences, while finding some other ambiguities that were spurious (such as suggesting erroneously that (1b) could be a question about how many different reasons you gave John in a single instance). Similarly, (2a) has both a de re and a de dicto reading, whereas (2b) has only a de re reading: 2a. How many books did Bill say that Mary should read? b. How many books did Bill explain why Mary should read? That is, (2a) can be asked either in a scenario where Bill has said "read 10 books!" or in a scenario where Bill has said "read Book A, Book B and Book C!" without necessarily counting the books himself. (2b), on the other hand, only has the second kind of interpretation. I've had mixed results with GPT-4 in this case (depending on exact choices of vocabulary, etc.), but it certainly makes some mistakes. For example, it says that (2b) can mean "John explained the reason for a certain number of books that Mary bought". As the sibling comment points out, it would not show very much if GTP-4 did correctly determine these ambiguities as it has had access to much more data than a child. You would also need to show that the same statistical techniques would work when applied to a realistic dataset.
      - stared 2y ago
        Thank you for providing these examples. I asked GPT4o, and it has no trouble with understanding 1a: https://chatgpt.com/share/daea469f-d823-45e2-9d6b-f6bea82a2623 https://chatgpt.com/share/daea469f-d823-45e2-9d6b-f6bea82a26... As a side note, my instinctive reading is on the telling frequency. Sure, one can make a garden path sentence, but (for my own ESL eyes and ears) it would be more straightforward to say, "Did you tell John how often he should take out the trash?" 2b does not feel right on its own (and I am not an AI). I can understand it, but it feels like reverse engineering rather than reading a normal sentence.
        foldr 2y ago
- foobarqux 2y ago
  First, Chomsky (and I) were talking about language acquisition in children where there aren't billions of examples, so it's completely irrelevant if some other system can do something, the question is how do humans do it. Second, there isn't any evidence that LLMs have captured grammar rules in any meaningful sense, just as they can't do addition or any other recursive computation.
- numpad0 2y ago
  Don't waste your time arguing Chomsky supporters, it's a cult. They keep wringing non-falsifiable theories out of lengthy hallucinations, but it all reveals itself to be just Trek physics later on. He's not just problematic in politics, he is in linguistics too, just the BS is harder to notice. I suppose his compiler theories were legit, and his pioneering spirit that lead to establishment of multiple fields of researches might be too, but leading theories he'd created are just as dubious as the first man made shed built on a discovered island would be. And the problem is not just that early speculations in a novel field are often wrong, it's that his supporters don't care. They'd regurgitate those scientific theories(tm) ad infinitum and waste resources for the whole humanity. So don't bother trying to fix them and making them motivated.