6 ms·

Please don't wildly speculate about technologies you clearly don't understand. "Does not strike me as a particularly high bar" means you're unfamiliar with how

Please don't wildly speculate about technologies you clearly don't understand. "Does not strike me as a particularly high bar" means you're unfamiliar with how these systems work at a deep level (by which I mean, you personally cannot sit down at a terminal and build one). So rather than -ahem- making things up and asking "what then?" -- please ask for textbook recommendations on these topics if you'd like to know more.

p1esk 6y ago
I guess we'll just have to wait and see what happens next year :) p.s. I built my first language model (LSTM based) back in 2014. Then I built a VAE based one. Then a GAN based. None of them were especially good, so I switched to music generation (this actually works pretty well). My most recent project is using sparse transformers for raw audio generation. Building novel DL models is literally in my job description. Please don't wildly speculate about strangers you meet on HN.
- inimino 6y ago
  Why just wait when you can bet?
  - p1esk 6y ago
    In order to bet, we would have to agree on evaluation criteria. A task like "respond to criticisms against its own argument in a coherent way" is difficult to evaluate. Turing test is also pretty vague, and some people already declared it passed many years ago: https://www.bbc.com/news/technology-27762088 https://www.bbc.com/news/technology-27762088
    - inimino 6y ago
      Yes, it would be difficult but not impossible to agree on clear criteria. We would also need an impartial third party to make a determination. The Turing test hasn't been passed if I'm the judge. Supposing I were unable to tell the difference between any AI system and a human interlocutor defending the same argument, at any time before 2023, I'd admit my prediction was wholly incorrect. I doubt very much I'll find a taker for this bet, however. The AI field has always been bigger on optimism than results, as we both know.
      - p1esk 6y ago
        Yes, I agree TT has not been passed. But probably starting later this year we will be seeing more and more claims it has. At first it will be clear it's not. Then not so clear, and then the goalposts will be moved again, so that when GPT-5 is announced and it is clearly capable of keeping a conversation, the reaction on HN will be the same as the current reaction to GPT-3: "meh".
        inimino 6y ago
        That seems like a pretty testable prediction. If I can't tell the difference between GPT-5 or whatever it is and an adult human native English speaker by the end of 2022, you'll win the bet.
        p1esk 6y ago
        How many questions will you need to determine if it's a bot? How many sessions, and what percentage of correct guesses will determine the outcome of the experiment?
        inimino 6y ago
        If it wasn't a bot but a human, we'd typically have an unbounded conversation, except by the bounds of politeness, until I was satisfied one way or the other, so that seems a reasonable protocol here, perhaps with some reasonable upper bound on the time spent (an hour?) to avoid putting a potentially unbounded commitment on human participants. (Actually, "We've been chatting since four, I have somewhere else to be" is a pretty good signal of humanity... and should help you see why you're not going to win this bet.) I wouldn't expect it to take more than a few minutes, and I would expect to be wrong in (much) less than 1% of sessions.
        p1esk 6y ago
        The more time you spent chatting the better your chances are to guess correctly. I admit that in 2 years the models might not be good enough to fool you for hours. If you put enough thought into it, especially knowing what the bot was trained on, you could devise a set of tricky questions which would expose it. However, I believe the models will be good enough to fool you during, say, a 20 question/response dialog. They will definitely be able to fool vast majority of unsuspecting humans. And they will definitely be able to keep track of conversation (remember what you said previously, and use it to construct responses to follow up questions).

p1esk 6y ago

I guess we'll just have to wait and see what happens next year :) p.s. I built my first language model (LSTM based) back in 2014. Then I built a VAE based one. Then a GAN based. None of them were especially good, so I switched to music generation (this actually works pretty well). My most recent project is using sparse transformers for raw audio generation. Building novel DL models is literally in my job description. Please don't wildly speculate about strangers you meet on HN.

inimino 6y ago
Why just wait when you can bet?
- p1esk 6y ago
  In order to bet, we would have to agree on evaluation criteria. A task like "respond to criticisms against its own argument in a coherent way" is difficult to evaluate. Turing test is also pretty vague, and some people already declared it passed many years ago: https://www.bbc.com/news/technology-27762088 https://www.bbc.com/news/technology-27762088
  - inimino 6y ago
    Yes, it would be difficult but not impossible to agree on clear criteria. We would also need an impartial third party to make a determination. The Turing test hasn't been passed if I'm the judge. Supposing I were unable to tell the difference between any AI system and a human interlocutor defending the same argument, at any time before 2023, I'd admit my prediction was wholly incorrect. I doubt very much I'll find a taker for this bet, however. The AI field has always been bigger on optimism than results, as we both know.
    - p1esk 6y ago
      Yes, I agree TT has not been passed. But probably starting later this year we will be seeing more and more claims it has. At first it will be clear it's not. Then not so clear, and then the goalposts will be moved again, so that when GPT-5 is announced and it is clearly capable of keeping a conversation, the reaction on HN will be the same as the current reaction to GPT-3: "meh".
      - inimino 6y ago
        That seems like a pretty testable prediction. If I can't tell the difference between GPT-5 or whatever it is and an adult human native English speaker by the end of 2022, you'll win the bet.
        p1esk 6y ago
        How many questions will you need to determine if it's a bot? How many sessions, and what percentage of correct guesses will determine the outcome of the experiment?
        inimino 6y ago
        If it wasn't a bot but a human, we'd typically have an unbounded conversation, except by the bounds of politeness, until I was satisfied one way or the other, so that seems a reasonable protocol here, perhaps with some reasonable upper bound on the time spent (an hour?) to avoid putting a potentially unbounded commitment on human participants. (Actually, "We've been chatting since four, I have somewhere else to be" is a pretty good signal of humanity... and should help you see why you're not going to win this bet.) I wouldn't expect it to take more than a few minutes, and I would expect to be wrong in (much) less than 1% of sessions.
        p1esk 6y ago
        The more time you spent chatting the better your chances are to guess correctly. I admit that in 2 years the models might not be good enough to fool you for hours. If you put enough thought into it, especially knowing what the bot was trained on, you could devise a set of tricky questions which would expose it. However, I believe the models will be good enough to fool you during, say, a 20 question/response dialog. They will definitely be able to fool vast majority of unsuspecting humans. And they will definitely be able to keep track of conversation (remember what you said previously, and use it to construct responses to follow up questions).