4 ms·

> It's way more important that CLIP learned to read (crudely) It learned to detect the features that make up the word "iPod", and match them against other feat

> It's way more important that CLIP learned to read (crudely) It learned to detect the features that make up the word "iPod", and match them against other features. I don't see that as being any different than detecting the features that make an apple, or a car or a face. It's just detecting the minimum number of features to distinguish the images in a training set. I'm generally disappointed with how breathlessly this forum gets excited about NN software. It doesn't 'get' images as a whole and compare them. There's no symbolics going on underneath, it's just picking up 'something' in an image and matching it with 'something else'. It's clearly nothing like how we see, let alone how we think or imagine. It's impressive and a useful innovation for sure, but it's not even on the ladder to our capabilities yet and I'm not convinced that it ever will be.

moyix 6y ago
To be clear, by "learned to read" – it actually learned to OCR text, unsupervised. That is pretty impressive, IMO. As for whether NNs are doing symbolic manipulation – I'm not so sure that they aren't. DALL-E, the sibling of CLIP, can do some things that IMO are hard to explain if there's nothing symbolic going on, like the examples in this post [1]. Of course we can't say it's doing what humans do (we don't know what humans do!) but it looks, to me, like more than pure pattern matching. nostalgebraist had some good thoughts on this (and actually so too did Gary Marcus well before him) [2]. [1] https://openai.com/blog/dall-e/ https://openai.com/blog/dall-e/ [2] https://nostalgebraist.tumblr.com/post/189965935059/human-psycholinguists-a-critical-appraisal https://nostalgebraist.tumblr.com/post/189965935059/human-ps...
- a1369209993 6y ago
  > by "learned to read" - it actually learned to OCR text, unsupervised. I think their claim is that it did not learn to OCR text, but merely to recognise certain specific images that happen to be things a human would interpret as specific bits of text.
  - the8472 6y ago
    From the CLIP article: > While CLIP’s zero-shot OCR performance is mixed, its semantic OCR representation is quite useful. When evaluated on the SST-2 NLP dataset rendered as images, a linear classifer on CLIP’s representation matches a CBoW model with direct access to the text. CLIP matches other models in text sentiment analysis when presented with image renderings of the text while the other model had direct access to the text. It also does MNIST. It might not be the best OCR software ever, but that's besides the point. It's huge advancement in generality on visual tasks and part of that generalization is some reading capability.
  - moyix 6y ago
    Yes, to be extra clear, if that is their claim, their claim is wrong.

moyix 6y ago

To be clear, by "learned to read" – it actually learned to OCR text, unsupervised. That is pretty impressive, IMO. As for whether NNs are doing symbolic manipulation – I'm not so sure that they aren't. DALL-E, the sibling of CLIP, can do some things that IMO are hard to explain if there's nothing symbolic going on, like the examples in this post [1]. Of course we can't say it's doing what humans do (we don't know what humans do!) but it looks, to me, like more than pure pattern matching. nostalgebraist had some good thoughts on this (and actually so too did Gary Marcus well before him) [2]. [1] https://openai.com/blog/dall-e/ https://openai.com/blog/dall-e/ [2] https://nostalgebraist.tumblr.com/post/189965935059/human-psycholinguists-a-critical-appraisal https://nostalgebraist.tumblr.com/post/189965935059/human-ps...

a1369209993 6y ago
> by "learned to read" - it actually learned to OCR text, unsupervised. I think their claim is that it did not learn to OCR text, but merely to recognise certain specific images that happen to be things a human would interpret as specific bits of text.
- the8472 6y ago
  From the CLIP article: > While CLIP’s zero-shot OCR performance is mixed, its semantic OCR representation is quite useful. When evaluated on the SST-2 NLP dataset rendered as images, a linear classifer on CLIP’s representation matches a CBoW model with direct access to the text. CLIP matches other models in text sentiment analysis when presented with image renderings of the text while the other model had direct access to the text. It also does MNIST. It might not be the best OCR software ever, but that's besides the point. It's huge advancement in generality on visual tasks and part of that generalization is some reading capability.
- moyix 6y ago
  Yes, to be extra clear, if that is their claim, their claim is wrong.