5 ms·

This piece would have been a lot better if it were maybe three paragraphs long. In summary: 1. Scott Alexander should have used an off-the-shelf benchmark like

This piece would have been a lot better if it were maybe three paragraphs long. In summary: 1. Scott Alexander should have used an off-the-shelf benchmark like Winoground instead of rolling his own five-question test. 2. He shouldn’t declare victory after cherry-picking good results from a small sample of questions.

robg 4y ago
3. And don’t test each example 10 times and conclude 1 correct guess equals success.
badloginagain 4y ago
I personally liked the anecdote about Clever Hans. I also learned there's a long history of AI skepticism, the root of which comes down to "Compositionality(?)"- and this wall of understanding meaning has vexed AI for decades. That would be lost in proposed short form summary.
lalaithion 4y ago
Scott didn't make up the rules, he agreed on them with another person who thought this would not happen in 3 years. Gary Marcus might have thought it was a bad bet, but someone was on the other side of it, and they presumably thought it was fair or they wouldn't have made it. The original terms of the bet: My proposed operationalization of this is that on June 1, 2025, if either if us can get access to the best image generating model at that time (I get to decide which), or convince someone else who has access to help us, we'll give it the following prompts: 1. A stained glass picture of a woman in a library with a raven on her shoulder with a key in its mouth 2. An oil painting of a man in a factory looking at a cat wearing a top hat 3. A digital art picture of a child riding a llama with a bell on its tail through a desert 4. A 3D render of an astronaut in space holding a fox wearing lipstick 5. Pixel art of a farmer in a cathedral holding a red basketball We generate 10 images for each prompt, just like DALL-E2 does. If at least one of the ten images has the scene correct in every particular on 3/5 prompts, I win, otherwise you do.
- skybrian 4y ago
  I think you got that wrong; Scott wrote the terms. (He wrote the comment [1] with those rules.) Someone in the comments agreed to them. Then he changed the terms because Imagen won't do people. I think that's cheating. [1] https://astralcodexten.substack.com/p/a-guide-to-asking-robots-to-design/comment/6945486 https://astralcodexten.substack.com/p/a-guide-to-asking-robo...
  - cwillu 4y ago
    Again, if the counter-party agrees to the terms and the changes, how is it cheating?
    - skybrian 4y ago
      It's not clear whether the counter-party agreed to the change. See: https://news.ycombinator.com/item?id=32858426 https://news.ycombinator.com/item?id=32858426
  - lalaithion 4y ago
    I think you missed the point of my comment. Yes, Scott wrote the comment containing that proposal. But my point was that it was an agreement. Two people who disagreed about AI agreed on the rules, so you can't accuse one of them of being unfair because you don't like the rules. Sure, you can say "that's a bad bet, Scott will obviously win", but you can't say "He shouldn’t declare victory after cherry-picking good results from a small sample of questions", because those terms were explicitly set in advance. The humans -> robots change is possibly dubious, yes. I don't think that it's super important, but if it were me, I wouldn't have posted the blog post as is. I would have waited until some AI passed all the prompts with humans, like it most certainly will in a year.
    - telotortium 4y ago
      Stable Diffusion will soon update to use the biggest CLIP model in existence, which may improve understanding of composition: https://news.ycombinator.com/edit?id=32858809 https://news.ycombinator.com/edit?id=32858809
nonameiguess 4y ago
For whatever reason, Gary doesn't even mention this, but from reading Scott's post, I don't think I agree that it even got 1/5, let alone 3/5. The bell is not on the llama's tail in any of the examples, though it is very close to the tail in one. The robot is either looking over the cat or in an unrelated direction, never at the cat. None of those basketball pictures shows a robot farmer. The fact that one may be wearing a hat doesn't make it a farmer. He says he's being generous because he believes it would have gotten a farmer more easily than robot farmer, which may be true, but a human artist would easily be able to depict a robot farmer. At least one other key to making a bet like this fair is that it needs to be arbitrated by a third party. He shouldn't get to decide himself if he won or not.
- kbelder 4y ago
  I agree with you, that 3/5 is stretching. This seems premature. But, at the rate we're seeing progress, I don't think there's any doubt at this point that top of the line models will be able to do all the proposed examples by June 2025. In fact, by June 2025 I bet that millions of people will be able to generate those images on their home computers.
  - aetherson 4y ago
    A lot of people in 2016 looked at the rapid progress of driverless cars in the few years prior and declared that there was no doubt we'd have full autonomy by 2022.
    - trention 4y ago
      >by 2022 Make that 2020. And in 2012 I was personally told by a startuper in the field that I would be able to buy L5 in 2017.
      - ben_w 4y ago
        In 2009 I was certain we'd have cars with no driving seat available for general purchase by 2018.
amusedcyclist 4y ago
- dang 4y ago
  Maybe so, but would you please stop posting unsubstantive and/or flamebait comments to HN, and please start following the site guidelines? We ban accounts that won't, for what ought to be obvious reasons. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

robg 4y ago

3. And don’t test each example 10 times and conclude 1 correct guess equals success.

badloginagain 4y ago

I personally liked the anecdote about Clever Hans. I also learned there's a long history of AI skepticism, the root of which comes down to "Compositionality(?)"- and this wall of understanding meaning has vexed AI for decades. That would be lost in proposed short form summary.

lalaithion 4y ago

Scott didn't make up the rules, he agreed on them with another person who thought this would not happen in 3 years. Gary Marcus might have thought it was a bad bet, but someone was on the other side of it, and they presumably thought it was fair or they wouldn't have made it. The original terms of the bet: My proposed operationalization of this is that on June 1, 2025, if either if us can get access to the best image generating model at that time (I get to decide which), or convince someone else who has access to help us, we'll give it the following prompts: 1. A stained glass picture of a woman in a library with a raven on her shoulder with a key in its mouth 2. An oil painting of a man in a factory looking at a cat wearing a top hat 3. A digital art picture of a child riding a llama with a bell on its tail through a desert 4. A 3D render of an astronaut in space holding a fox wearing lipstick 5. Pixel art of a farmer in a cathedral holding a red basketball We generate 10 images for each prompt, just like DALL-E2 does. If at least one of the ten images has the scene correct in every particular on 3/5 prompts, I win, otherwise you do.

skybrian 4y ago
I think you got that wrong; Scott wrote the terms. (He wrote the comment [1] with those rules.) Someone in the comments agreed to them. Then he changed the terms because Imagen won't do people. I think that's cheating. [1] https://astralcodexten.substack.com/p/a-guide-to-asking-robots-to-design/comment/6945486 https://astralcodexten.substack.com/p/a-guide-to-asking-robo...
- cwillu 4y ago
  Again, if the counter-party agrees to the terms and the changes, how is it cheating?
  - skybrian 4y ago
    It's not clear whether the counter-party agreed to the change. See: https://news.ycombinator.com/item?id=32858426 https://news.ycombinator.com/item?id=32858426
- lalaithion 4y ago
  I think you missed the point of my comment. Yes, Scott wrote the comment containing that proposal. But my point was that it was an agreement. Two people who disagreed about AI agreed on the rules, so you can't accuse one of them of being unfair because you don't like the rules. Sure, you can say "that's a bad bet, Scott will obviously win", but you can't say "He shouldn’t declare victory after cherry-picking good results from a small sample of questions", because those terms were explicitly set in advance. The humans -> robots change is possibly dubious, yes. I don't think that it's super important, but if it were me, I wouldn't have posted the blog post as is. I would have waited until some AI passed all the prompts with humans, like it most certainly will in a year.
  - telotortium 4y ago
    Stable Diffusion will soon update to use the biggest CLIP model in existence, which may improve understanding of composition: https://news.ycombinator.com/edit?id=32858809 https://news.ycombinator.com/edit?id=32858809

nonameiguess 4y ago

For whatever reason, Gary doesn't even mention this, but from reading Scott's post, I don't think I agree that it even got 1/5, let alone 3/5. The bell is not on the llama's tail in any of the examples, though it is very close to the tail in one. The robot is either looking over the cat or in an unrelated direction, never at the cat. None of those basketball pictures shows a robot farmer. The fact that one may be wearing a hat doesn't make it a farmer. He says he's being generous because he believes it would have gotten a farmer more easily than robot farmer, which may be true, but a human artist would easily be able to depict a robot farmer. At least one other key to making a bet like this fair is that it needs to be arbitrated by a third party. He shouldn't get to decide himself if he won or not.

kbelder 4y ago
I agree with you, that 3/5 is stretching. This seems premature. But, at the rate we're seeing progress, I don't think there's any doubt at this point that top of the line models will be able to do all the proposed examples by June 2025. In fact, by June 2025 I bet that millions of people will be able to generate those images on their home computers.
- aetherson 4y ago
  A lot of people in 2016 looked at the rapid progress of driverless cars in the few years prior and declared that there was no doubt we'd have full autonomy by 2022.
  - trention 4y ago
    >by 2022 Make that 2020. And in 2012 I was personally told by a startuper in the field that I would be able to buy L5 in 2017.
    - ben_w 4y ago
      In 2009 I was certain we'd have cars with no driving seat available for general purchase by 2018.

amusedcyclist 4y ago

dang 4y ago
Maybe so, but would you please stop posting unsubstantive and/or flamebait comments to HN, and please start following the site guidelines? We ban accounts that won't, for what ought to be obvious reasons. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html