5 ms·

Chomsky describes why the word identification method described in the article can’t work in English (though a more elaborate method is one of the few successes

Chomsky describes why the word identification method described in the article can’t work in English (though a more elaborate method is one of the few successes he knows in applying statistical methods to language): https://youtu.be/92GOS1VIGxY https://youtu.be/92GOS1VIGxY The arguments against universal grammar are no good either: For example even though children may hear lots of examples it isn’t nearly enough to derive a hierarchical grammar. It also doesn’t explain why language is hierarchical (just like it doesn’t explain why we can’t speak and hear like a modem)

thfuran 2y ago
Meaning what, at a few kilobaud?
new299 2y ago
Anyone have a link to the work he cites?
- deleted 2y ago
  [deleted]
- foldr 2y ago
  I haven't watched the video, but probably a reference to this and subsequent work along the same lines: https://aclanthology.org/W04-1307.pdf https://aclanthology.org/W04-1307.pdf https://www.ling.upenn.edu/~ycharles/papers/quick.pdf https://www.ling.upenn.edu/~ycharles/papers/quick.pdf https://sites.socsci.uci.edu/~lpearl/courses/readings/Yang2004_AcqData_UGStatisticsOrBoth.pdf https://sites.socsci.uci.edu/~lpearl/courses/readings/Yang20...
stared 2y ago
Once you go beyond Markov chains and into vector models (Transformers or even LSTMs), next —-token-- syllable prediction can capture grammar.
- foldr 2y ago
  That some aspects of grammar can be captured by statistical analysis was never really in dispute. The OP is slightly confused about the hierarchy part. Chomsky never said that you couldn't discover by statistical analysis that languages have a hierarchical structure. He rather said that a baby, based on the data it has available, would not be able to determine statistically that certain rules of grammar are defined over this structure rather than over the linear sequence of words.
  - stared 2y ago
    I mean, contrary to Chomsky, that certainly advanced statistical models are able to capture all grammar - at least, nit worse than ordinary humans. It feels that Chomsky’s understanding of ML stopped at the level of Markov chains.
    - foldr 2y ago
      Is there any work demonstrating this? For example, how do statistical models capture adjunct/argument asymmetries in extraction?
      - stared 2y ago
        Attention is one of the core parts of transformer architecture, so I would be surprised if they have any trouble understanding this asymmetry. Could you provide a testable hypothesis? I would be happy to test it on GPT4.
        foldr 2y ago
        Sure, here are a couple of examples of ECP violations removing ambiguities. 1a. How often did you tell John that he should take out the trash? b. How often did you tell John why he should take out the trash? (1a) can either be a question about frequency of telling or frequency of trash disposal, whereas (1b) can only be a question about frequency of telling. I asked GPT-4 to explain how each sentence was ambiguous and it seemed to entirely miss the embedded readings (the ones about frequency of trash disposal) for both sentences, while finding some other ambiguities that were spurious (such as suggesting erroneously that (1b) could be a question about how many different reasons you gave John in a single instance). Similarly, (2a) has both a de re and a de dicto reading, whereas (2b) has only a de re reading: 2a. How many books did Bill say that Mary should read? b. How many books did Bill explain why Mary should read? That is, (2a) can be asked either in a scenario where Bill has said "read 10 books!" or in a scenario where Bill has said "read Book A, Book B and Book C!" without necessarily counting the books himself. (2b), on the other hand, only has the second kind of interpretation. I've had mixed results with GPT-4 in this case (depending on exact choices of vocabulary, etc.), but it certainly makes some mistakes. For example, it says that (2b) can mean "John explained the reason for a certain number of books that Mary bought". As the sibling comment points out, it would not show very much if GTP-4 did correctly determine these ambiguities as it has had access to much more data than a child. You would also need to show that the same statistical techniques would work when applied to a realistic dataset.
contingencies 2y ago
Honestly, the first thing I did was ctrl+F and chomsky, and as there was no reference I didn't bother reading the article. Thanks for inserting some credit where credit is due. This is coming from a background of serious interest in computational linguistics in the late 1990s... I'm aware Chomsky's since dethroned, but the contribution was massive. I am greatly disinterested in these days of popular generative ML, where LLMs ~= hall of mirrors(internet-scale abject kerfuffery) and then people guffaw in awe of the remix.

thfuran 2y ago

Meaning what, at a few kilobaud?

new299 2y ago

Anyone have a link to the work he cites?

deleted 2y ago
[deleted]
foldr 2y ago
I haven't watched the video, but probably a reference to this and subsequent work along the same lines: https://aclanthology.org/W04-1307.pdf https://aclanthology.org/W04-1307.pdf https://www.ling.upenn.edu/~ycharles/papers/quick.pdf https://www.ling.upenn.edu/~ycharles/papers/quick.pdf https://sites.socsci.uci.edu/~lpearl/courses/readings/Yang2004_AcqData_UGStatisticsOrBoth.pdf https://sites.socsci.uci.edu/~lpearl/courses/readings/Yang20...

stared 2y ago

Once you go beyond Markov chains and into vector models (Transformers or even LSTMs), next —-token-- syllable prediction can capture grammar.

foldr 2y ago
That some aspects of grammar can be captured by statistical analysis was never really in dispute. The OP is slightly confused about the hierarchy part. Chomsky never said that you couldn't discover by statistical analysis that languages have a hierarchical structure. He rather said that a baby, based on the data it has available, would not be able to determine statistically that certain rules of grammar are defined over this structure rather than over the linear sequence of words.
- stared 2y ago
  I mean, contrary to Chomsky, that certainly advanced statistical models are able to capture all grammar - at least, nit worse than ordinary humans. It feels that Chomsky’s understanding of ML stopped at the level of Markov chains.
  - foldr 2y ago
    Is there any work demonstrating this? For example, how do statistical models capture adjunct/argument asymmetries in extraction?
    - stared 2y ago
      Attention is one of the core parts of transformer architecture, so I would be surprised if they have any trouble understanding this asymmetry. Could you provide a testable hypothesis? I would be happy to test it on GPT4.
      - foldr 2y ago
        Sure, here are a couple of examples of ECP violations removing ambiguities. 1a. How often did you tell John that he should take out the trash? b. How often did you tell John why he should take out the trash? (1a) can either be a question about frequency of telling or frequency of trash disposal, whereas (1b) can only be a question about frequency of telling. I asked GPT-4 to explain how each sentence was ambiguous and it seemed to entirely miss the embedded readings (the ones about frequency of trash disposal) for both sentences, while finding some other ambiguities that were spurious (such as suggesting erroneously that (1b) could be a question about how many different reasons you gave John in a single instance). Similarly, (2a) has both a de re and a de dicto reading, whereas (2b) has only a de re reading: 2a. How many books did Bill say that Mary should read? b. How many books did Bill explain why Mary should read? That is, (2a) can be asked either in a scenario where Bill has said "read 10 books!" or in a scenario where Bill has said "read Book A, Book B and Book C!" without necessarily counting the books himself. (2b), on the other hand, only has the second kind of interpretation. I've had mixed results with GPT-4 in this case (depending on exact choices of vocabulary, etc.), but it certainly makes some mistakes. For example, it says that (2b) can mean "John explained the reason for a certain number of books that Mary bought". As the sibling comment points out, it would not show very much if GTP-4 did correctly determine these ambiguities as it has had access to much more data than a child. You would also need to show that the same statistical techniques would work when applied to a realistic dataset.

contingencies 2y ago

Honestly, the first thing I did was ctrl+F and chomsky, and as there was no reference I didn't bother reading the article. Thanks for inserting some credit where credit is due. This is coming from a background of serious interest in computational linguistics in the late 1990s... I'm aware Chomsky's since dethroned, but the contribution was massive. I am greatly disinterested in these days of popular generative ML, where LLMs ~= hall of mirrors(internet-scale abject kerfuffery) and then people guffaw in awe of the remix.