3 ms·

Drawing graphs is probably one of the worst comparisons one can do in terms of evaluating these models. They seem to be trained to generate either photorealisti

Drawing graphs is probably one of the worst comparisons one can do in terms of evaluating these models. They seem to be trained to generate either photorealistic or stylized images.

sberens 4y ago
I had personally never seen this side of the models, so I wanted to share this finding. I agree that they were trained more on artistic images, but I was still surprised with how bad they generalized to a more theoretical(?) context.
- mgraczyk 4y ago
  I think the path forward for something like this is models that learn to execute python code and incorporate the results into their outputs. There are already projects that can generate correct matplotlib calls for prompts like yours, but I don't think we are to the point where those python outputs can be automatically combined with a diffusion model for style or whatever.
- post-it 4y ago
  It's not about it being theoretical, it's moreso that the language model is still far more simplistic than our own, and struggles with anything but the most basic relations between nouns. The "horse riding an astronaut" post is a good example of this.[0] [0] https://garymarcus.substack.com/p/horse-rides-astronaut https://garymarcus.substack.com/p/horse-rides-astronaut
  - ytdytvhxgydvhh 4y ago
    See also “Shirt without stripes” https://github.com/elsamuko/Shirt-without-Stripes https://github.com/elsamuko/Shirt-without-Stripes
  - throwaman123 4y ago
    It’s funny to see this comment on HN, because it gets refuted so many times by now you should be able to punch into google and find it. https://twitter.com/Plinz/status/1529013919682994176 https://twitter.com/Plinz/status/1529013919682994176
    - cycomanic 4y ago
      Did you even read the linked article? They even cite the picture you link, it still proves their point though these models have no understanding of the language (like Google claimed).
    - post-it 4y ago
      I just asked Stable Diffusion to generate 10 images of "horse riding an astronaut" and 10/10 were of an astronaut riding a horse.
    - amanuonsense 4y ago
      This is what it should look like: https://www.bdaddik.com/en/comics-collectible-postcards/2967-postcard-lucky-luke-lucky-luke-lifting-jolly-jumper-10x15cm-5054021675186.html https://www.bdaddik.com/en/comics-collectible-postcards/2967... Modulo s/Lucky Luke/astronaut/g Note that the image above should be in Dall-E's training set. So it's seen how a horse rides a human. No excuses there.
      - optimalsolver 4y ago
        The comment you're replying to shows Dall-E doing exactly this. Did you click the link?
        amanuonsense 4y ago
        Isn't yours just a bad faith comment? Why would I not have clicked the link?
- noduerme 4y ago
  One thing they're not very good at is deducing spatial relationships. Concepts like "above", "inside" and "behind", or "after". I'd say the prompts you gave make sense to a human who is thinking of a visual progression from left to right. I bet you could write a few copilot prompts to generate code which would draw a graph like this, though.
- sanderjd 4y ago
  I was surprised while reading your article how good the AIs did. I think it's fascinating that your intuition was that these tools would be able to do a good job of drawing a graph based on a description of it...
amanuonsense 4y ago
After reading your comment I asked Stable Diffusion to create photorealistic images of a graph with three lines (similar to the smallest prompt in the article). Here's the results of three attempts with slightly different prompts: https://imgur.com/a/FqQT2Mk https://imgur.com/a/FqQT2Mk As you'll see, Stable Diffusion a) is perfectly capable of drawing graphs, and, b) completely incapable of drawing a simple graph with three lines as prompted.

Drawing graphs is probably one of the worst comparisons one can do in terms of evaluating these models. They seem to be trained to generate either photorealistic or stylized images.

sberens 4y ago

I had personally never seen this side of the models, so I wanted to share this finding. I agree that they were trained more on artistic images, but I was still surprised with how bad they generalized to a more theoretical(?) context.

mgraczyk 4y ago
I think the path forward for something like this is models that learn to execute python code and incorporate the results into their outputs. There are already projects that can generate correct matplotlib calls for prompts like yours, but I don't think we are to the point where those python outputs can be automatically combined with a diffusion model for style or whatever.
post-it 4y ago
It's not about it being theoretical, it's moreso that the language model is still far more simplistic than our own, and struggles with anything but the most basic relations between nouns. The "horse riding an astronaut" post is a good example of this.[0] [0] https://garymarcus.substack.com/p/horse-rides-astronaut https://garymarcus.substack.com/p/horse-rides-astronaut
- ytdytvhxgydvhh 4y ago
  See also “Shirt without stripes” https://github.com/elsamuko/Shirt-without-Stripes https://github.com/elsamuko/Shirt-without-Stripes
- throwaman123 4y ago
  It’s funny to see this comment on HN, because it gets refuted so many times by now you should be able to punch into google and find it. https://twitter.com/Plinz/status/1529013919682994176 https://twitter.com/Plinz/status/1529013919682994176
  - cycomanic 4y ago
    Did you even read the linked article? They even cite the picture you link, it still proves their point though these models have no understanding of the language (like Google claimed).
  - post-it 4y ago
    I just asked Stable Diffusion to generate 10 images of "horse riding an astronaut" and 10/10 were of an astronaut riding a horse.
  - amanuonsense 4y ago
    This is what it should look like: https://www.bdaddik.com/en/comics-collectible-postcards/2967-postcard-lucky-luke-lucky-luke-lifting-jolly-jumper-10x15cm-5054021675186.html https://www.bdaddik.com/en/comics-collectible-postcards/2967... Modulo s/Lucky Luke/astronaut/g Note that the image above should be in Dall-E's training set. So it's seen how a horse rides a human. No excuses there.
    - optimalsolver 4y ago
      The comment you're replying to shows Dall-E doing exactly this. Did you click the link?
      - amanuonsense 4y ago
        Isn't yours just a bad faith comment? Why would I not have clicked the link?
noduerme 4y ago
One thing they're not very good at is deducing spatial relationships. Concepts like "above", "inside" and "behind", or "after". I'd say the prompts you gave make sense to a human who is thinking of a visual progression from left to right. I bet you could write a few copilot prompts to generate code which would draw a graph like this, though.
sanderjd 4y ago
I was surprised while reading your article how good the AIs did. I think it's fascinating that your intuition was that these tools would be able to do a good job of drawing a graph based on a description of it...

amanuonsense 4y ago

After reading your comment I asked Stable Diffusion to create photorealistic images of a graph with three lines (similar to the smallest prompt in the article). Here's the results of three attempts with slightly different prompts: https://imgur.com/a/FqQT2Mk https://imgur.com/a/FqQT2Mk As you'll see, Stable Diffusion a) is perfectly capable of drawing graphs, and, b) completely incapable of drawing a simple graph with three lines as prompted.