5 ms·

Scott didn't make up the rules, he agreed on them with another person who thought this would not happen in 3 years. Gary Marcus might have thought it was a bad

Scott didn't make up the rules, he agreed on them with another person who thought this would not happen in 3 years. Gary Marcus might have thought it was a bad bet, but someone was on the other side of it, and they presumably thought it was fair or they wouldn't have made it. The original terms of the bet: My proposed operationalization of this is that on June 1, 2025, if either if us can get access to the best image generating model at that time (I get to decide which), or convince someone else who has access to help us, we'll give it the following prompts: 1. A stained glass picture of a woman in a library with a raven on her shoulder with a key in its mouth 2. An oil painting of a man in a factory looking at a cat wearing a top hat 3. A digital art picture of a child riding a llama with a bell on its tail through a desert 4. A 3D render of an astronaut in space holding a fox wearing lipstick 5. Pixel art of a farmer in a cathedral holding a red basketball We generate 10 images for each prompt, just like DALL-E2 does. If at least one of the ten images has the scene correct in every particular on 3/5 prompts, I win, otherwise you do.

skybrian 4y ago
I think you got that wrong; Scott wrote the terms. (He wrote the comment [1] with those rules.) Someone in the comments agreed to them. Then he changed the terms because Imagen won't do people. I think that's cheating. [1] https://astralcodexten.substack.com/p/a-guide-to-asking-robots-to-design/comment/6945486 https://astralcodexten.substack.com/p/a-guide-to-asking-robo...
- cwillu 4y ago
  Again, if the counter-party agrees to the terms and the changes, how is it cheating?
  - skybrian 4y ago
    It's not clear whether the counter-party agreed to the change. See: https://news.ycombinator.com/item?id=32858426 https://news.ycombinator.com/item?id=32858426
- lalaithion 4y ago
  I think you missed the point of my comment. Yes, Scott wrote the comment containing that proposal. But my point was that it was an agreement. Two people who disagreed about AI agreed on the rules, so you can't accuse one of them of being unfair because you don't like the rules. Sure, you can say "that's a bad bet, Scott will obviously win", but you can't say "He shouldn’t declare victory after cherry-picking good results from a small sample of questions", because those terms were explicitly set in advance. The humans -> robots change is possibly dubious, yes. I don't think that it's super important, but if it were me, I wouldn't have posted the blog post as is. I would have waited until some AI passed all the prompts with humans, like it most certainly will in a year.
  - telotortium 4y ago
    Stable Diffusion will soon update to use the biggest CLIP model in existence, which may improve understanding of composition: https://news.ycombinator.com/edit?id=32858809 https://news.ycombinator.com/edit?id=32858809
    - TOMDM 4y ago
      Is it the largest CLIP model or the largest open source CLIP model?
      - dhruval 4y ago
        The latter
  - mannykannot 4y ago
    I feel your point, in turn, misses the point of the article. Yes, given that someone accepted the terms and those terms were met, then Alexander won the bet, no question. That particular fact about the bet, however, does nothing to counter Marcus' criticisms of Alexander's methodology and his claims of significant progress on the compositionality problem.
    - thaumasiotes 4y ago
- SteveDR 4y ago
  Cheating? Thatd make sense if the bet were about the future of products and ethics. Weren’t they trying to predict the future of the state of the art technology?
  - skybrian 4y ago
    It depends on what you mean by "technology" and "exists." A research project at Google intentionally won't render people. Maybe it could render people, theoretically, but without evidence, we don't know how well.
techbio 4y ago
These are all the same artwork.
- WJW 4y ago
  The terms of the bet don't refer to any specific artwork, only to the best image generating model. Hence, you are correct but it does not matter for the outcome of the bet under discussion.
Natsu 4y ago
Honestly, the whole thing makes me wonder if we can use this to generate CAPTCHAs. I don't think a human would have trouble picking out which image was the lightbulb surrounding leaves, but apparently AI still does.
- thayne 4y ago
  The tricky part there is you would need a really big sample of such prompts, that adversaries don't have access to. And since AI can't generate such images yet, you can't randomly generate them.
thayne 4y ago
So what? Someone else agreeing to the terms of his bet doesn't mean it is a good evaluation of AIs capabilities. And the terms of the bet say that he is cherry-picking the results that meet the prompt. The article isn't saying that Scott didn't win his bet, it is saying that winning that bet doesn't really say that Imagen has solved the compositionality problem.
spullara 4y ago
I'm on your side of the bet for 2023.
moonchrome 4y ago
These kind of abstract things are pointless tests - when do you need a stained glass picture of a woman in a library with a raven on her shoulder with a key in its mouth ? You're likely to accept some wildly inaccurate things in those images because the subject is so abstract. A more practical use case is "bicycle, branded x, with y frame shape, with an adult male, 40-50, riding down hill in mountain road in spring" - now that's something I can use as stock photo. This example is very specific - but insert whatever product you want in whatever scenario you need it. Here it becomes important that you understand features of the objects you're drawing to avoid making colossal mistakes, and you're going to notice if the model doesn't understand it right away. Painting abstract portraits and random art is fun but you're willing to accept so much as correct that it's not a very useful measure of model quality (personally).
- shmeeed 4y ago
  Funny enough, this prompt wasn't originally engineered by Scott for the contest, but for an intended art piece, and the symbolism does check out: https://astralcodexten.substack.com/p/a-guide-to-asking-robots-to-design https://astralcodexten.substack.com/p/a-guide-to-asking-robo... The Eleventh Virtue: Scholarship My plan for this one was Alexandra Elbakyan (the Sci-Hub woman) in a library, with the Sci-Hub mascot (a raven with a key in its mouth).

skybrian 4y ago

I think you got that wrong; Scott wrote the terms. (He wrote the comment [1] with those rules.) Someone in the comments agreed to them. Then he changed the terms because Imagen won't do people. I think that's cheating. [1] https://astralcodexten.substack.com/p/a-guide-to-asking-robots-to-design/comment/6945486 https://astralcodexten.substack.com/p/a-guide-to-asking-robo...

cwillu 4y ago
Again, if the counter-party agrees to the terms and the changes, how is it cheating?
- skybrian 4y ago
  It's not clear whether the counter-party agreed to the change. See: https://news.ycombinator.com/item?id=32858426 https://news.ycombinator.com/item?id=32858426
lalaithion 4y ago
I think you missed the point of my comment. Yes, Scott wrote the comment containing that proposal. But my point was that it was an agreement. Two people who disagreed about AI agreed on the rules, so you can't accuse one of them of being unfair because you don't like the rules. Sure, you can say "that's a bad bet, Scott will obviously win", but you can't say "He shouldn’t declare victory after cherry-picking good results from a small sample of questions", because those terms were explicitly set in advance. The humans -> robots change is possibly dubious, yes. I don't think that it's super important, but if it were me, I wouldn't have posted the blog post as is. I would have waited until some AI passed all the prompts with humans, like it most certainly will in a year.
- telotortium 4y ago
  Stable Diffusion will soon update to use the biggest CLIP model in existence, which may improve understanding of composition: https://news.ycombinator.com/edit?id=32858809 https://news.ycombinator.com/edit?id=32858809
  - TOMDM 4y ago
    Is it the largest CLIP model or the largest open source CLIP model?
    - dhruval 4y ago
      The latter
- mannykannot 4y ago
  I feel your point, in turn, misses the point of the article. Yes, given that someone accepted the terms and those terms were met, then Alexander won the bet, no question. That particular fact about the bet, however, does nothing to counter Marcus' criticisms of Alexander's methodology and his claims of significant progress on the compositionality problem.
  - thaumasiotes 4y ago
SteveDR 4y ago
Cheating? Thatd make sense if the bet were about the future of products and ethics. Weren’t they trying to predict the future of the state of the art technology?
- skybrian 4y ago
  It depends on what you mean by "technology" and "exists." A research project at Google intentionally won't render people. Maybe it could render people, theoretically, but without evidence, we don't know how well.

techbio 4y ago

These are all the same artwork.

WJW 4y ago
The terms of the bet don't refer to any specific artwork, only to the best image generating model. Hence, you are correct but it does not matter for the outcome of the bet under discussion.

Natsu 4y ago

Honestly, the whole thing makes me wonder if we can use this to generate CAPTCHAs. I don't think a human would have trouble picking out which image was the lightbulb surrounding leaves, but apparently AI still does.

thayne 4y ago
The tricky part there is you would need a really big sample of such prompts, that adversaries don't have access to. And since AI can't generate such images yet, you can't randomly generate them.

thayne 4y ago

So what? Someone else agreeing to the terms of his bet doesn't mean it is a good evaluation of AIs capabilities. And the terms of the bet say that he is cherry-picking the results that meet the prompt. The article isn't saying that Scott didn't win his bet, it is saying that winning that bet doesn't really say that Imagen has solved the compositionality problem.

spullara 4y ago

I'm on your side of the bet for 2023.

moonchrome 4y ago

These kind of abstract things are pointless tests - when do you need a stained glass picture of a woman in a library with a raven on her shoulder with a key in its mouth ? You're likely to accept some wildly inaccurate things in those images because the subject is so abstract. A more practical use case is "bicycle, branded x, with y frame shape, with an adult male, 40-50, riding down hill in mountain road in spring" - now that's something I can use as stock photo. This example is very specific - but insert whatever product you want in whatever scenario you need it. Here it becomes important that you understand features of the objects you're drawing to avoid making colossal mistakes, and you're going to notice if the model doesn't understand it right away. Painting abstract portraits and random art is fun but you're willing to accept so much as correct that it's not a very useful measure of model quality (personally).

shmeeed 4y ago
Funny enough, this prompt wasn't originally engineered by Scott for the contest, but for an intended art piece, and the symbolism does check out: https://astralcodexten.substack.com/p/a-guide-to-asking-robots-to-design https://astralcodexten.substack.com/p/a-guide-to-asking-robo... The Eleventh Virtue: Scholarship My plan for this one was Alexandra Elbakyan (the Sci-Hub woman) in a library, with the Sci-Hub mascot (a raven with a key in its mouth).