5 ms·

I wonder if there’s any reason this wouldn’t also apply in reverse: could a corpus of textual training data from which the role of a hand can be derived (includ

I wonder if there’s any reason this wouldn’t also apply in reverse: could a corpus of textual training data from which the role of a hand can be derived (including the fact that it naturally has five fingers) help DALL-E reliably render hands correctly?

lumost 4y ago
Almost certainly. There is a vast corpus of text which references hands and what they are used for. Likewise there is a vast corpus of images which include hands. A sufficiently advanced statistical model can infer that any image with 6 fingers is a fabrication provided that it has learned how to classify fingers and has enough linguistic knowledge. Contrast this situation with a primarily image based model which will struggle to differentiate what situations detailed hands appear in compared with a patterned rug. There is an interesting “takeoff” point where models are able to generate enough “interesting” data to train themselves. Asking chatGPT for ml training data on various tasks makes me think we are close to that inflection point.
- noah_buddy 4y ago
  Can’t wait for AI to tell people with atypical bodies that they don’t exist.
  - lumost 4y ago
    Definitely a bias, chatGPT will likely believe that only individuals with figures similar to models are real.
- IIAOPSW 4y ago
  The hands on the clock pointed to 15 past 2... Dalle/GPT: that's not right, there should be 5 fingers on this clock.
- fnordpiglet 4y ago
  I think it would say “My name is Inigo Montoya. You Killed my Father. Prepare to die.”
gwern 4y ago
One of the biggest improvements to text models like Image/Parti which are much better than Stable Diffusion is that they typically either train a much larger text model from scratch, or plug in a very good pre-trained text model like T5. This instantly upgrades their images. It turns out that the image models already understand the low-level details of images well, and it's high-level reasoning/composition that is screwing their samples up - which is due to the poor text embedding they receive as a summary. (You can't generate a good image of 'a blue ball left of a red ball' if your text embedding is confused what 'left' is, no matter how high quality you are able to generate 'blue ball' or 'red ball' individually.) So, I would not be surprised if hands are improved by simply using a much better text model. (And why not plug in GPT-3 or PaLM for even better results...) That said, hands are also like text-inside-images in being small, highly variable, and intrinsically difficult. And we know that the larger models solve text rendering simply by scaling up an OOM or 2 past SD. (You can see this from the text samples in the papers, and also more recently from Stability's Deep Floyd samples they've been teasing on Twitter.) So either way, scaling will probably solve hands soon.

lumost 4y ago

Almost certainly. There is a vast corpus of text which references hands and what they are used for. Likewise there is a vast corpus of images which include hands. A sufficiently advanced statistical model can infer that any image with 6 fingers is a fabrication provided that it has learned how to classify fingers and has enough linguistic knowledge. Contrast this situation with a primarily image based model which will struggle to differentiate what situations detailed hands appear in compared with a patterned rug. There is an interesting “takeoff” point where models are able to generate enough “interesting” data to train themselves. Asking chatGPT for ml training data on various tasks makes me think we are close to that inflection point.

noah_buddy 4y ago
Can’t wait for AI to tell people with atypical bodies that they don’t exist.
- lumost 4y ago
  Definitely a bias, chatGPT will likely believe that only individuals with figures similar to models are real.
IIAOPSW 4y ago
The hands on the clock pointed to 15 past 2... Dalle/GPT: that's not right, there should be 5 fingers on this clock.
fnordpiglet 4y ago
I think it would say “My name is Inigo Montoya. You Killed my Father. Prepare to die.”

gwern 4y ago

One of the biggest improvements to text models like Image/Parti which are much better than Stable Diffusion is that they typically either train a much larger text model from scratch, or plug in a very good pre-trained text model like T5. This instantly upgrades their images. It turns out that the image models already understand the low-level details of images well, and it's high-level reasoning/composition that is screwing their samples up - which is due to the poor text embedding they receive as a summary. (You can't generate a good image of 'a blue ball left of a red ball' if your text embedding is confused what 'left' is, no matter how high quality you are able to generate 'blue ball' or 'red ball' individually.) So, I would not be surprised if hands are improved by simply using a much better text model. (And why not plug in GPT-3 or PaLM for even better results...) That said, hands are also like text-inside-images in being small, highly variable, and intrinsically difficult. And we know that the larger models solve text rendering simply by scaling up an OOM or 2 past SD. (You can see this from the text samples in the papers, and also more recently from Stability's Deep Floyd samples they've been teasing on Twitter.) So either way, scaling will probably solve hands soon.