3 ms·

I often hear on places like here that Scott Alexander is interesting and deep and insightful. But then I see bits and pieces like this. This blogger doesn't nee

I often hear on places like here that Scott Alexander is interesting and deep and insightful. But then I see bits and pieces like this. This blogger doesn't need to go into some deep analysis of compositionality to go "You came up with a 5 question test and decided 1 answer out of 10 attempts would be a pass". We've gone from 90%+s in imagenet to this as a pass mark? It's like sure we can dissect all the statistical risks of this, but why bother? It's self evident bullshit. You might as well have just posted a link to Scott Alexander's original blog claiming victory with just "Lol ok". Just post a screenshot of the phrase "An oil painting of a robot in a factory looking at a cat wearing a top hat", show the pictures of a robot near a cat that has a top hat, not in a warehouse, and say "lol ok."

phreeza 4y ago
I feel like he has kind of lost his spark a bit, but he does draw an interesting group of commenters. Similar to hacker news in that regard, sometimes the linked articles are a bit mundane but there is gold in the comments.
ravi-delia 4y ago
This blog misses the point; he made a bet, and the people on the other side also accepted the terms. Nowhere did Alexander claim composition was a solved problem, just that the terms of the bet were satisfied. Generative models are still bad at composition, but claiming they will literally never improve requires some amount of additional evidence
- Viliam1234 4y ago
  Exactly. The bet wasn't that AI will do composition correctly all the time, but that it will do composition correctly sometimes. That is how both sides understood it. The analogy with a student at exam misses the point. If you do art -- even as a human artist -- you do not need a 100% success rate. A 10% success rate is okay if you are willing to simply throw away the remaining 90% of the pictures. If you have an AI that at a click of a button can generate 10 beautiful pictures, 1 of them containing exactly what you wanted, that just means you need to make two clicks in order to get the picture you wanted. That is an awesome thing.
comex 4y ago
He never said that compositionality was solved. He said: > Without wanting to claim that Imagen has fully mastered compositionality, I think it represents a significant enough improvement to win the bet, and to provide some evidence that simple scaling and normal progress are enough for compositionality gains. Given these gains, it would surprise me (though by no means be impossible) if image model skill plateaued at this level rather than continuing to improve. It seems plausible to me that Imagen is consistently a little better than DALL-E at compositionality. The stained glass pictures are always stained glass. The top hat is always on the cat rather than the person/robot (that may be partially because robots are less likely to wear top hats, but I just tried the robot version of the prompt in Stable Diffusion and it usually puts the top hat on the robot). The astronaut and farmer examples are less clear, but they're not as obviously misinterpreted as the DALL-E versions (which tends to put the lipstick on the astronaut and the farmer in front of a cathedral). To be fair, I'm not sure how much difference that really makes; I would be pretty shocked if a newer model wasn't a little better, and it could still hypothetically be approaching an asymptote. Also, it would have been better to get a larger sample size. But he did set that low standard ahead of time; someone bet against him that the state of the art wouldn't get even a little better, in a significantly longer period of time. And, seemingly, it did.

phreeza 4y ago

I feel like he has kind of lost his spark a bit, but he does draw an interesting group of commenters. Similar to hacker news in that regard, sometimes the linked articles are a bit mundane but there is gold in the comments.

ravi-delia 4y ago

This blog misses the point; he made a bet, and the people on the other side also accepted the terms. Nowhere did Alexander claim composition was a solved problem, just that the terms of the bet were satisfied. Generative models are still bad at composition, but claiming they will literally never improve requires some amount of additional evidence

Viliam1234 4y ago
Exactly. The bet wasn't that AI will do composition correctly all the time, but that it will do composition correctly sometimes. That is how both sides understood it. The analogy with a student at exam misses the point. If you do art -- even as a human artist -- you do not need a 100% success rate. A 10% success rate is okay if you are willing to simply throw away the remaining 90% of the pictures. If you have an AI that at a click of a button can generate 10 beautiful pictures, 1 of them containing exactly what you wanted, that just means you need to make two clicks in order to get the picture you wanted. That is an awesome thing.

comex 4y ago

He never said that compositionality was solved. He said: > Without wanting to claim that Imagen has fully mastered compositionality, I think it represents a significant enough improvement to win the bet, and to provide some evidence that simple scaling and normal progress are enough for compositionality gains. Given these gains, it would surprise me (though by no means be impossible) if image model skill plateaued at this level rather than continuing to improve. It seems plausible to me that Imagen is consistently a little better than DALL-E at compositionality. The stained glass pictures are always stained glass. The top hat is always on the cat rather than the person/robot (that may be partially because robots are less likely to wear top hats, but I just tried the robot version of the prompt in Stable Diffusion and it usually puts the top hat on the robot). The astronaut and farmer examples are less clear, but they're not as obviously misinterpreted as the DALL-E versions (which tends to put the lipstick on the astronaut and the farmer in front of a cathedral). To be fair, I'm not sure how much difference that really makes; I would be pretty shocked if a newer model wasn't a little better, and it could still hypothetically be approaching an asymptote. Also, it would have been better to get a larger sample size. But he did set that low standard ahead of time; someone bet against him that the state of the art wouldn't get even a little better, in a significantly longer period of time. And, seemingly, it did.