37 ms·

The post states, "When we train GPT-2 on images unrolled into long sequences of pixels, which we call iGPT, we find that the model appears to understand 2-D ima

The post states, "When we train GPT-2 on images unrolled into long sequences of pixels, which we call iGPT, we find that the model appears to understand 2-D image characteristics such as object appearance and category." This, to me (former philosophy student) feels like a very low bar for "understanding." Could the model _explain_ 2D image characteristics, or can it only generate them? I'm sure this debate will rage on for a while, but when it comes to intelligence, I believe we ought to be more rigorous with our use of the word.

markchen90 6y ago
Author here. You're absolutely right that "understanding" is a fuzzy word. As you pointed out, part of reason we hold this belief is that the model can generate diverse samples and successfully complete out-of-distribution inputs. But the other part is that the model learns (without labels) useful features for classifying objects. Would be very interesting to test it on a broader set of datasets which measure other 2D image characteristics.
xvedejas 6y ago
I work in computer vision; "understanding" here is meant to mean there is a statistical understanding. The model better represents the distribution of things that look like real images. I must say these examples are incredibly good at generating coherent scenes compared to any similar attempts I've seen. I do think it represents an advance in the that kind of understanding. I think your definition of "understanding" is more the realm of AI, and why so many people in my field sigh when the term "AI" is used when we're really just talking about ML. But then again, you could absolutely in the current day train an ML model to look at images and then produce a natural language "explanation" of the image. It might not be able to make leaps in deductive logic, but it would be able to explain more than just a list of what is in the image. Is this "understanding"? Maybe that question is philosophy.
inertiatic 6y ago
A convolutional net learns kernels that most people would say display some understanding over the domain of images as well. At points they might even serve as explanations. I think taking "understanding" to be "able to explain rigorously" is the highest bar possible here. A lot of people understand concepts that they are unable to explain. Animals understand a lot but lack a way of explaining things as well.

markchen90 6y ago

Author here. You're absolutely right that "understanding" is a fuzzy word. As you pointed out, part of reason we hold this belief is that the model can generate diverse samples and successfully complete out-of-distribution inputs. But the other part is that the model learns (without labels) useful features for classifying objects. Would be very interesting to test it on a broader set of datasets which measure other 2D image characteristics.

xvedejas 6y ago

I work in computer vision; "understanding" here is meant to mean there is a statistical understanding. The model better represents the distribution of things that look like real images. I must say these examples are incredibly good at generating coherent scenes compared to any similar attempts I've seen. I do think it represents an advance in the that kind of understanding. I think your definition of "understanding" is more the realm of AI, and why so many people in my field sigh when the term "AI" is used when we're really just talking about ML. But then again, you could absolutely in the current day train an ML model to look at images and then produce a natural language "explanation" of the image. It might not be able to make leaps in deductive logic, but it would be able to explain more than just a list of what is in the image. Is this "understanding"? Maybe that question is philosophy.

inertiatic 6y ago

A convolutional net learns kernels that most people would say display some understanding over the domain of images as well. At points they might even serve as explanations. I think taking "understanding" to be "able to explain rigorously" is the highest bar possible here. A lot of people understand concepts that they are unable to explain. Animals understand a lot but lack a way of explaining things as well.