4 ms·

> If you flip a penny 5 times and get 5 heads, you need to calculate that the chance of getting that particular outcome is 1 in 32. If you conduct the experimen

> If you flip a penny 5 times and get 5 heads, you need to calculate that the chance of getting that particular outcome is 1 in 32. If you conduct the experiment often enough, you’re going to get that, but it doesn’t mean that much. If you get 3/5 as Alexander did, when he prematurely declared victory, you don’t have much evidence of anything at all. This doesn’t make much sense. The task at hand is in no way equivalent in difficulty to flipping a coin. This is kind of like saying, “if you beat Usain Bolt in a race 3/5 times, that doesn’t mean anything; it’s like getting 3/5 coin flips to be heads.”

Tenoke 4y ago
While I'm generally very unsympathetic to Marcus' anti-AI arguments at this point, this critique makes some sense. If e.g. the model is just combining the features at random, you'd expect it to combine them the right way over enough tries. It isn't that simple, and I don't believe it matters as this is hardly the peak model we'll get but in isolation his objection is valid.
- ALittleLight 4y ago
  I think you would need to do some kind of analysis. For example, if your prompt was "red ball on top of blue cube" and you want to know if the results come from chance you'd need to know the likelihood of the model putting the red ball on top of the blue cube by chance. There are maybe four relative positions for red ball to blue cube - beside, above, below, in, around. Are they each equally likely? I would try to get a collection of prompts like "red ball and blue cube" or "an empty plane containing only a red ball and a blue cube" and so on - try to come up with 20 or 30 of these. Then, generate 100 images for each prompt. Next, see how likely it is for a red ball to randomly be on top of a blue cube when it was not directed to be. After gathering some baseline data we could then test three prompts. "Red ball on top of blue cube" and "Red ball beside blue cube" and "Red ball below blue cube". Generate 100 or 1000 images for each of these prompts. Count respective orientations. Then, decide whether red ball being on top of blue cube is more likely than the baseline when the specific direction is given and whether it is less likely when contrary directions are given.
  - ummonk 4y ago
    It might understand that there is a cube, there is a ball, the scene has red and blue parts, and there is a vertical placement (“on top of”). In that case it would get 1 out of 4 images right.
    - ALittleLight 4y ago
      Yes, it might, but that should be settled by experiment rather than speculation.
- adamsmith143 4y ago
  The point is that the probability space of potential generated images is enormous so a 3/5 success rate represents an absurdly unlikely probability of being due to chance.
  - peteradio 4y ago
    That would depend on how you define the phase space.
thayne 4y ago
Sure, the probabilities are different (although as far as I know we don't what those probabilities actually are), but the same principle applies. To take your Usain Bolt example, if you won 3 out of 5 races against him, that might just be because it was an off day for him, and not because you are actually faster than him. If you won 300 out of 500 races done in various circumstances on different days, then that is much more conclusive that you are faster than him. And this bet was even worse than that, because im each of the five tests, the best out of 10 results is picked.
- meowface 4y ago
  >To take your Usain Bolt example, if you won 3 out of 5 races against him, that might just be because it was an off day for him, and not because you are actually faster than him. It shows you're probably very competitive with him, though, barring some special circumstance where he says he's suffering from an illness or whatever. You can't compare either racing Usain Bolt or generating complex images with flipping coins. The conditions of this bet demonstrate that AIs are getting better at correctly understanding the specific intentions of prompts when generating images, even if it doesn't show they're anywhere near human-level understanding.
  - raviparikh 4y ago
    Exactly my point. Maybe I got lucky, but to have gotten that lucky in the first place, I'd have to have world-class running speed at all. Generating image compositions sounds fairly difficult to do by random. If you took 3 different objects and randomly placed them in a square canvas, the odds that they'd look reasonably placed seem pretty low. So 3/5 correct seems like a non-trivial accomplishment.
    - thayne 4y ago
      > I'd have to have world-class running speed at all. Or he was really sick or something. > So 3/5 correct seems like a non-trivial accomplishment. It's definitely a non-trivial accomplishment. And it does show that Imagen can get it right sometimes. But with a sample size of 5, you certainly don't have enough data to say it can consistently get descriptions like these right 3/5 of the time. And the question at hand isn't "can it draw what I asked it to instead of random garbage" it's "can it combine multiple parts of a sentence in the correct way", which, assuming that determining the correct components is already a solved problem, doesn't have as many degrees of freedom. For example in the "astronaut riding a horse" example, if it has half of the results with an astronaut riding a horse, and half with a horse riding an astronaut, it clearly doesn't understand how it is composed, but you still have a decent chance of getting the right image. Especially if you take 10 samples and pick the best one.

Tenoke 4y ago

While I'm generally very unsympathetic to Marcus' anti-AI arguments at this point, this critique makes some sense. If e.g. the model is just combining the features at random, you'd expect it to combine them the right way over enough tries. It isn't that simple, and I don't believe it matters as this is hardly the peak model we'll get but in isolation his objection is valid.

ALittleLight 4y ago
I think you would need to do some kind of analysis. For example, if your prompt was "red ball on top of blue cube" and you want to know if the results come from chance you'd need to know the likelihood of the model putting the red ball on top of the blue cube by chance. There are maybe four relative positions for red ball to blue cube - beside, above, below, in, around. Are they each equally likely? I would try to get a collection of prompts like "red ball and blue cube" or "an empty plane containing only a red ball and a blue cube" and so on - try to come up with 20 or 30 of these. Then, generate 100 images for each prompt. Next, see how likely it is for a red ball to randomly be on top of a blue cube when it was not directed to be. After gathering some baseline data we could then test three prompts. "Red ball on top of blue cube" and "Red ball beside blue cube" and "Red ball below blue cube". Generate 100 or 1000 images for each of these prompts. Count respective orientations. Then, decide whether red ball being on top of blue cube is more likely than the baseline when the specific direction is given and whether it is less likely when contrary directions are given.
- ummonk 4y ago
  It might understand that there is a cube, there is a ball, the scene has red and blue parts, and there is a vertical placement (“on top of”). In that case it would get 1 out of 4 images right.
  - ALittleLight 4y ago
    Yes, it might, but that should be settled by experiment rather than speculation.
adamsmith143 4y ago
The point is that the probability space of potential generated images is enormous so a 3/5 success rate represents an absurdly unlikely probability of being due to chance.
- peteradio 4y ago
  That would depend on how you define the phase space.

thayne 4y ago

Sure, the probabilities are different (although as far as I know we don't what those probabilities actually are), but the same principle applies. To take your Usain Bolt example, if you won 3 out of 5 races against him, that might just be because it was an off day for him, and not because you are actually faster than him. If you won 300 out of 500 races done in various circumstances on different days, then that is much more conclusive that you are faster than him. And this bet was even worse than that, because im each of the five tests, the best out of 10 results is picked.

meowface 4y ago
>To take your Usain Bolt example, if you won 3 out of 5 races against him, that might just be because it was an off day for him, and not because you are actually faster than him. It shows you're probably very competitive with him, though, barring some special circumstance where he says he's suffering from an illness or whatever. You can't compare either racing Usain Bolt or generating complex images with flipping coins. The conditions of this bet demonstrate that AIs are getting better at correctly understanding the specific intentions of prompts when generating images, even if it doesn't show they're anywhere near human-level understanding.
- raviparikh 4y ago
  Exactly my point. Maybe I got lucky, but to have gotten that lucky in the first place, I'd have to have world-class running speed at all. Generating image compositions sounds fairly difficult to do by random. If you took 3 different objects and randomly placed them in a square canvas, the odds that they'd look reasonably placed seem pretty low. So 3/5 correct seems like a non-trivial accomplishment.
  - thayne 4y ago
    > I'd have to have world-class running speed at all. Or he was really sick or something. > So 3/5 correct seems like a non-trivial accomplishment. It's definitely a non-trivial accomplishment. And it does show that Imagen can get it right sometimes. But with a sample size of 5, you certainly don't have enough data to say it can consistently get descriptions like these right 3/5 of the time. And the question at hand isn't "can it draw what I asked it to instead of random garbage" it's "can it combine multiple parts of a sentence in the correct way", which, assuming that determining the correct components is already a solved problem, doesn't have as many degrees of freedom. For example in the "astronaut riding a horse" example, if it has half of the results with an astronaut riding a horse, and half with a horse riding an astronaut, it clearly doesn't understand how it is composed, but you still have a decent chance of getting the right image. Especially if you take 10 samples and pick the best one.