6 ms·

Why just wait when you can bet?

p1esk 6y ago
In order to bet, we would have to agree on evaluation criteria. A task like "respond to criticisms against its own argument in a coherent way" is difficult to evaluate. Turing test is also pretty vague, and some people already declared it passed many years ago: https://www.bbc.com/news/technology-27762088 https://www.bbc.com/news/technology-27762088
- inimino 6y ago
  Yes, it would be difficult but not impossible to agree on clear criteria. We would also need an impartial third party to make a determination. The Turing test hasn't been passed if I'm the judge. Supposing I were unable to tell the difference between any AI system and a human interlocutor defending the same argument, at any time before 2023, I'd admit my prediction was wholly incorrect. I doubt very much I'll find a taker for this bet, however. The AI field has always been bigger on optimism than results, as we both know.
  - p1esk 6y ago
    Yes, I agree TT has not been passed. But probably starting later this year we will be seeing more and more claims it has. At first it will be clear it's not. Then not so clear, and then the goalposts will be moved again, so that when GPT-5 is announced and it is clearly capable of keeping a conversation, the reaction on HN will be the same as the current reaction to GPT-3: "meh".
    - inimino 6y ago
      That seems like a pretty testable prediction. If I can't tell the difference between GPT-5 or whatever it is and an adult human native English speaker by the end of 2022, you'll win the bet.
      - p1esk 6y ago
        How many questions will you need to determine if it's a bot? How many sessions, and what percentage of correct guesses will determine the outcome of the experiment?
        inimino 6y ago
        If it wasn't a bot but a human, we'd typically have an unbounded conversation, except by the bounds of politeness, until I was satisfied one way or the other, so that seems a reasonable protocol here, perhaps with some reasonable upper bound on the time spent (an hour?) to avoid putting a potentially unbounded commitment on human participants. (Actually, "We've been chatting since four, I have somewhere else to be" is a pretty good signal of humanity... and should help you see why you're not going to win this bet.) I wouldn't expect it to take more than a few minutes, and I would expect to be wrong in (much) less than 1% of sessions.
        p1esk 6y ago
        The more time you spent chatting the better your chances are to guess correctly. I admit that in 2 years the models might not be good enough to fool you for hours. If you put enough thought into it, especially knowing what the bot was trained on, you could devise a set of tricky questions which would expose it. However, I believe the models will be good enough to fool you during, say, a 20 question/response dialog. They will definitely be able to fool vast majority of unsuspecting humans. And they will definitely be able to keep track of conversation (remember what you said previously, and use it to construct responses to follow up questions).
        inimino 6y ago
        How will the bot answer "when and where were you born, and how do you know?" How will it answer "what color, besides red, best communicates the flavor of a strawberry, and why?". How will it answer "What historical figure does my communication style make you think of most, and why?", or "Which of your family members comes to your mind first?" or "What do you think the context was in which the following poem was written?". I don't need to know what it was trained on to win this bet, and 20 open-ended questions is more than enough. Between the vast majority of unsuspecting humans and me there is a considerable gap. Mind the gap!

In order to bet, we would have to agree on evaluation criteria. A task like "respond to criticisms against its own argument in a coherent way" is difficult to evaluate. Turing test is also pretty vague, and some people already declared it passed many years ago: https://www.bbc.com/news/technology-27762088 https://www.bbc.com/news/technology-27762088

inimino 6y ago
Yes, it would be difficult but not impossible to agree on clear criteria. We would also need an impartial third party to make a determination. The Turing test hasn't been passed if I'm the judge. Supposing I were unable to tell the difference between any AI system and a human interlocutor defending the same argument, at any time before 2023, I'd admit my prediction was wholly incorrect. I doubt very much I'll find a taker for this bet, however. The AI field has always been bigger on optimism than results, as we both know.
- p1esk 6y ago
  Yes, I agree TT has not been passed. But probably starting later this year we will be seeing more and more claims it has. At first it will be clear it's not. Then not so clear, and then the goalposts will be moved again, so that when GPT-5 is announced and it is clearly capable of keeping a conversation, the reaction on HN will be the same as the current reaction to GPT-3: "meh".
  - inimino 6y ago
    That seems like a pretty testable prediction. If I can't tell the difference between GPT-5 or whatever it is and an adult human native English speaker by the end of 2022, you'll win the bet.
    - p1esk 6y ago
      How many questions will you need to determine if it's a bot? How many sessions, and what percentage of correct guesses will determine the outcome of the experiment?
      - inimino 6y ago
        If it wasn't a bot but a human, we'd typically have an unbounded conversation, except by the bounds of politeness, until I was satisfied one way or the other, so that seems a reasonable protocol here, perhaps with some reasonable upper bound on the time spent (an hour?) to avoid putting a potentially unbounded commitment on human participants. (Actually, "We've been chatting since four, I have somewhere else to be" is a pretty good signal of humanity... and should help you see why you're not going to win this bet.) I wouldn't expect it to take more than a few minutes, and I would expect to be wrong in (much) less than 1% of sessions.
        p1esk 6y ago
        The more time you spent chatting the better your chances are to guess correctly. I admit that in 2 years the models might not be good enough to fool you for hours. If you put enough thought into it, especially knowing what the bot was trained on, you could devise a set of tricky questions which would expose it. However, I believe the models will be good enough to fool you during, say, a 20 question/response dialog. They will definitely be able to fool vast majority of unsuspecting humans. And they will definitely be able to keep track of conversation (remember what you said previously, and use it to construct responses to follow up questions).
        inimino 6y ago
        How will the bot answer "when and where were you born, and how do you know?" How will it answer "what color, besides red, best communicates the flavor of a strawberry, and why?". How will it answer "What historical figure does my communication style make you think of most, and why?", or "Which of your family members comes to your mind first?" or "What do you think the context was in which the following poem was written?". I don't need to know what it was trained on to win this bet, and 20 open-ended questions is more than enough. Between the vast majority of unsuspecting humans and me there is a considerable gap. Mind the gap!