5 ms·

Let's consider the supreme court handshake problem, but say there is an unwritten social law that forces judges to only ever initiate a handshake with judges le

Let's consider the supreme court handshake problem, but say there is an unwritten social law that forces judges to only ever initiate a handshake with judges less senior than themselves. If the seniority of two particular judges happened to be exactly equal, they would not shake hands at all. Let's assume (I don't know if it is actually true or not) that in the history of the supreme court, there have never been two judges of the exact same seniority. In that case, a model learned from handshake data would not include the slightest hint of this unwritten social law. I think what Chomsky is saying is that if we do not understand the generative principle behind any data, we cannot possibly know what circumstance might completely invalidate our model. There may not be a way to smooth this out. Language understanding, contrary to things like speech recognition, does not lend itself very well to smoothing.

netcan 10y ago
If this is unwritten (I take this to mean a piece of culture people practice without ever formalizing), how would the judges deal with the novel scenario? That rule will not have existed until it starts being practiced.
- fauigerzigerk 10y ago
  The judges would simply apply the existing rule as it exists in their mind, but there is no trace of that state of mind in the handshake dataset. Yes the machine could guess. It could even guess correctly. But I think what Chomsky means is that we have no valid scientific reason to believe that it would.
  - netcan 10y ago
    If the machine is faced with this novel scenario before the judges, guesses, then that becomes the cultural norm.... does the machine win?
ilitirit 10y ago
> If the seniority of two particular judges happened to be exactly equal, they would not shake hands at all. Somewhat similar to Buridan's Ass. ... an ass that is equally hungry and thirsty is placed precisely midway between a stack of hay and a pail of water. Since the paradox assumes the ass will always go to whichever is closer, it will die of both hunger and thirst since it cannot make any rational decision to choose one over the other. https://en.wikipedia.org/wiki/Buridan%27s_ass https://en.wikipedia.org/wiki/Buridan%27s_ass
lordnacho 10y ago
Can't you learn this rule by generating (simulating) the behaviour with various hypotheses? So you'd have a model where handshakes are dependent on the relative heights, and it would fail because some senior judge who is short goes around and initiates. And then you'd come to the correct one, if the data is there for it. Like a hypothesis that's never refuted. Of course then there's noise. If it's a general rule, but on casual Fridays...
- gbrown 10y ago
  Yep, check out approximate Bayesian computing.
- lukeasrodgers 10y ago
  I think part of the problem with this approach is that there are infinitely many hypotheses (due to the infinitely generative nature of language) that are compatible with the behaviour, so to test and refute them all would be impossible. E.g. "only initiate handshakes with your juniors" is compatible with "only initiate handshakes with your juniors who have not been to Mars" etc. This example is silly and may seem trivial, but I think it illustrates the problem of trying to derive a principle or rule simply from observed behaviour.
  - lordnacho 10y ago
    But isn't that the nature of scientific hypotheses? They are tenuous, only valid until you come across some data that tells you "aha, there's another piece to this rule". The classic would be special relativity vs old Newtonian. If you have a limited dataset containing the actual generating data, say heights, seniority, age, hair color, day of the week, gender, etc, can't you end up discovering at least the parts that are exposed in the data?
    - YeGoblynQueenne 10y ago
      >> But isn't that the nature of scientific hypotheses? I guess scientists don't guess completely at random though, do they? They use some sort of heuristics, or their instinct (I've no idea what is, no) to decide which hypotheses are worth pursuing. Most statistical models aren't very good at taking background knowledge like that into account. Not to mention most are also sensitive to "noise" (which is to say, useful information that they can't make use of).
YeGoblynQueenne 10y ago
The way Norvig tells it, language lends itself perfectly well to smoothing and even Chomsky's favourite absurd phrase "colorless green ideas sleep furiously" is assigned a (very small but non-zero) probability in some statistical models of English. Truth is you can't do much with statistical models of language without some sort of way to account for what's missing from your data, which is always most of language. On the other hand, anything you might do is never going to be enough when that's the case: that you're missing the majority of language from your training data.
- fauigerzigerk 10y ago
  Some NLP tasks do indeed lend themselves well to smoothing, but I was thinking of language understanding tasks like question answering where changing or misunderstanding a single word in an entire novel can easily make a correct answer completely incorrect. I agree with your second paragraph, and if I understand Chomsky correctly, that is part of why he argues in favor of a generative grammar. I can't say that I completely understand how such a grammar would be linked to semantics and experience though.
  - YeGoblynQueenne 10y ago
    Ah, OK, sorry for misunderstanding.
    - mcguire 10y ago
      Was that a pun? Out of curiosity, how does Chomsky's generative model account for language understanding?
      - foobarqux 10y ago
        It demonstrates that human languages all have a certain structure, so it must be an innate faculty of human beings (as opposed to acquired). A neural network could not tell you that. It also suggests further avenues of inquiry (Why that particular structure?)
foobarqux 10y ago
> I think what Chomsky is saying is that if we do not understand the generative principle behind any data, we cannot possibly know what circumstance might completely invalidate our model. There may not be a way to smooth this out. No, what Chomsky is saying is that if you don't know the generative principle behind the data, then you don't know the generative principle behind the data. You don't understand a system just because you can faithfully predict it.

netcan 10y ago

If this is unwritten (I take this to mean a piece of culture people practice without ever formalizing), how would the judges deal with the novel scenario? That rule will not have existed until it starts being practiced.

fauigerzigerk 10y ago
The judges would simply apply the existing rule as it exists in their mind, but there is no trace of that state of mind in the handshake dataset. Yes the machine could guess. It could even guess correctly. But I think what Chomsky means is that we have no valid scientific reason to believe that it would.
- netcan 10y ago
  If the machine is faced with this novel scenario before the judges, guesses, then that becomes the cultural norm.... does the machine win?

ilitirit 10y ago

> If the seniority of two particular judges happened to be exactly equal, they would not shake hands at all. Somewhat similar to Buridan's Ass. ... an ass that is equally hungry and thirsty is placed precisely midway between a stack of hay and a pail of water. Since the paradox assumes the ass will always go to whichever is closer, it will die of both hunger and thirst since it cannot make any rational decision to choose one over the other. https://en.wikipedia.org/wiki/Buridan%27s_ass https://en.wikipedia.org/wiki/Buridan%27s_ass

lordnacho 10y ago

Can't you learn this rule by generating (simulating) the behaviour with various hypotheses? So you'd have a model where handshakes are dependent on the relative heights, and it would fail because some senior judge who is short goes around and initiates. And then you'd come to the correct one, if the data is there for it. Like a hypothesis that's never refuted. Of course then there's noise. If it's a general rule, but on casual Fridays...

gbrown 10y ago
Yep, check out approximate Bayesian computing.
lukeasrodgers 10y ago
I think part of the problem with this approach is that there are infinitely many hypotheses (due to the infinitely generative nature of language) that are compatible with the behaviour, so to test and refute them all would be impossible. E.g. "only initiate handshakes with your juniors" is compatible with "only initiate handshakes with your juniors who have not been to Mars" etc. This example is silly and may seem trivial, but I think it illustrates the problem of trying to derive a principle or rule simply from observed behaviour.
- lordnacho 10y ago
  But isn't that the nature of scientific hypotheses? They are tenuous, only valid until you come across some data that tells you "aha, there's another piece to this rule". The classic would be special relativity vs old Newtonian. If you have a limited dataset containing the actual generating data, say heights, seniority, age, hair color, day of the week, gender, etc, can't you end up discovering at least the parts that are exposed in the data?
  - YeGoblynQueenne 10y ago
    >> But isn't that the nature of scientific hypotheses? I guess scientists don't guess completely at random though, do they? They use some sort of heuristics, or their instinct (I've no idea what is, no) to decide which hypotheses are worth pursuing. Most statistical models aren't very good at taking background knowledge like that into account. Not to mention most are also sensitive to "noise" (which is to say, useful information that they can't make use of).

YeGoblynQueenne 10y ago

The way Norvig tells it, language lends itself perfectly well to smoothing and even Chomsky's favourite absurd phrase "colorless green ideas sleep furiously" is assigned a (very small but non-zero) probability in some statistical models of English. Truth is you can't do much with statistical models of language without some sort of way to account for what's missing from your data, which is always most of language. On the other hand, anything you might do is never going to be enough when that's the case: that you're missing the majority of language from your training data.

fauigerzigerk 10y ago
Some NLP tasks do indeed lend themselves well to smoothing, but I was thinking of language understanding tasks like question answering where changing or misunderstanding a single word in an entire novel can easily make a correct answer completely incorrect. I agree with your second paragraph, and if I understand Chomsky correctly, that is part of why he argues in favor of a generative grammar. I can't say that I completely understand how such a grammar would be linked to semantics and experience though.
- YeGoblynQueenne 10y ago
  Ah, OK, sorry for misunderstanding.
  - mcguire 10y ago
    Was that a pun? Out of curiosity, how does Chomsky's generative model account for language understanding?
    - foobarqux 10y ago
      It demonstrates that human languages all have a certain structure, so it must be an innate faculty of human beings (as opposed to acquired). A neural network could not tell you that. It also suggests further avenues of inquiry (Why that particular structure?)

foobarqux 10y ago

> I think what Chomsky is saying is that if we do not understand the generative principle behind any data, we cannot possibly know what circumstance might completely invalidate our model. There may not be a way to smooth this out. No, what Chomsky is saying is that if you don't know the generative principle behind the data, then you don't know the generative principle behind the data. You don't understand a system just because you can faithfully predict it.