3 ms·

Usually when people say "ML is just curve fitting" they mean to continue with something like "so it will never be able to compete with humans." The interesting

Usually when people say "ML is just curve fitting" they mean to continue with something like "so it will never be able to compete with humans." The interesting thing to me about the secret language is that it seems to imply that when DALL-E fit words to concepts, it created extrapolations in its curve fit that are more extreme than the actual training samples, ie. its fit has out-of-domain extrema. So there are letter sequences that are more "a whale on the moon" than the actual text "a whale on the moon." Linguistic superstimulus.

visarga 4y ago
Yes, I can confirm that's how I read the "just curve fitting" bit. Regarding the gibberish word to image issue - CLIP uses a text transformer trained by contrastive matching to images. That means it's different from GPT, where it trains to predict the probability of the next word. GPT would easily tell apart gibberish words from real words, or incorrect syntax because they would be low probability sequences. CLIP text transformer doesn't do that because of the task formulation, not because of an intrinsic limitation. It's not so mysterious after realising they could have used a different approach to have both the text embedding and filter out gibberish if they wanted. A good analogy would be a Rorschach test - show an OOD image to a human asking him to caption it. They will still say something about the image, just like DALL-E will draw a fake word. It's because the human is expected to generate a phrase no matter if the image makes sense or not, and DALL-E has a similar demand. The task formulation explains the result. The mapping from nonsense word to image is explained by the continuous embedding space of the prompt and the ability to generate images from noise of the diffusion model. Any point in the embedding space, even random ones, fall closer to some concepts and further from other concepts. The lucky concept most similar to the random embedding would trigger the image generation.
deltaonefour 4y ago
Usually except I went on to elaborate that curve fitting was essentially what intelligence was. If mr. Genius here read my post more carefully he wouldn't have had to reveal how immature he was with those comments.

visarga 4y ago

Yes, I can confirm that's how I read the "just curve fitting" bit. Regarding the gibberish word to image issue - CLIP uses a text transformer trained by contrastive matching to images. That means it's different from GPT, where it trains to predict the probability of the next word. GPT would easily tell apart gibberish words from real words, or incorrect syntax because they would be low probability sequences. CLIP text transformer doesn't do that because of the task formulation, not because of an intrinsic limitation. It's not so mysterious after realising they could have used a different approach to have both the text embedding and filter out gibberish if they wanted. A good analogy would be a Rorschach test - show an OOD image to a human asking him to caption it. They will still say something about the image, just like DALL-E will draw a fake word. It's because the human is expected to generate a phrase no matter if the image makes sense or not, and DALL-E has a similar demand. The task formulation explains the result. The mapping from nonsense word to image is explained by the continuous embedding space of the prompt and the ability to generate images from noise of the diffusion model. Any point in the embedding space, even random ones, fall closer to some concepts and further from other concepts. The lucky concept most similar to the random embedding would trigger the image generation.

deltaonefour 4y ago

Usually except I went on to elaborate that curve fitting was essentially what intelligence was. If mr. Genius here read my post more carefully he wouldn't have had to reveal how immature he was with those comments.