6 ms·
The SVG created for the first prompt is valid but is a garbage image.
by lifeisgood99 2y ago
The SVG created for the first prompt is valid but is a garbage image.
- simonw 2y agoYeah, it didn't do very well on that one. The best I've had from a local model there was from QwQ: https://simonwillison.net/2024/Nov/27/qwq/ https://simonwillison.net/2024/Nov/27/qwq/
- refulgentis 2y agoFor context, pelican riding a bicycle: https://imgur.com/a/2nhm0XM https://imgur.com/a/2nhm0XM Copied SVG from gist into figma, added dark gray #444444 background, exported as PNG 1x.
- bentcorner 2y agoIn general I've had poor results with LLMs generating pictures using text instructions (in my case I've tried to get them to generate pictures using plots in KQL). They work but the pictures are very very basic. I'd be interested for any LLM emitting any kind of text-to-picture instructions to get results that are beyond a kindergartner-cardboard-cutout levels of art.
- mycall 2y agoI've had success with LLMs producing mermaid.js or three.js output but that is a different use case.
- simonw 2y agoThat's why I use the SVG pelican riding a bicycle thing as a benchmark: it's a deliberately absurd and extremely difficult task.
- Teever 2y agoI'm really glad that I see someone else doing something similar. I had the epiphany a while ago that if LLMs can interpret textual instructions to draw a picture and output the design in another textual format that this a strong indicator that they're more than just stochastic parrots. My personal test has been "A horse eating apples next to a tree" but the deliberate absurdity of your example is a much more useful test. Do you know if this is a recognized technique that people use to study LLMs?
- simonw 2y agoI've seen people using "draw a unicorn using tikz" https://adamkdean.co.uk/posts/gpt-unicorn-a-daily-exploration-of-gpt-4s-image-generation-capabilities https://adamkdean.co.uk/posts/gpt-unicorn-a-daily-exploratio...
- girvo 2y agoThat came, IIRC, from one of the OpenAI or Microsoft people (Sebastian Bubeck); it was recounted in an NPR podcast "Greetings from Earth" https://www.thisamericanlife.org/803/transcript https://www.thisamericanlife.org/803/transcript
- krackers 2y agoIt's in this presentation https://www.youtube.com/watch?v=qbIk7-JPB2c https://www.youtube.com/watch?v=qbIk7-JPB2c The most significant part I took away is that when safety "alignment" was done the ability plummeted. So that really makes me wonder how much better these models would be if they weren't lobotomized to prevent them from saying bad words.
- int_19h 2y agoI did some experiments of my own after this paper, but letting GPT-4 run wild, picking its own scene. It wanted to draw a boat on a lake, and I also asked it to throw in some JS animations, so it made the sun set: https://int19h.org/chatgpt/lakeside/index.html https://int19h.org/chatgpt/lakeside/index.html One interesting thing that I found out while doing this is that if you ask GPT-4 to produce SVG suitable for use in HTML, it will often just generate base64-encoded data: URIs directly. Which do contain valid SVG inside as requested.
- memhole 2y agoNot sure if this counts. I recently went from description of a screenshot of graph to generate pandas code and plot from description. Conceptually it was accurate. I don’t think it reflects any understanding. But to go from screenshot to conceptually accurate and working code was impressive.
- accrual 2y agoAppreciate your rapid analysis of new models, Simon. Have any models you've tested performed well on the pelican SVG task?
- simonw 2y agogemini-exp-1206 is my new favorite: https://simonwillison.net/2024/Dec/6/gemini-exp-1206/ https://simonwillison.net/2024/Dec/6/gemini-exp-1206/ Claude 3.5 Sonnet is in second place: https://github.com/simonw/pelican-bicycle?tab=readme-ov-file#claude-35-sonnet-2024-06-20 https://github.com/simonw/pelican-bicycle?tab=readme-ov-file...
- carbocation 2y agoThe gemini result is great. I modified your prompt to encourage more detail ("Generate an SVG of a pelican riding a bicycle. The degree of detail should be surprisingly high and should spark delight for the viewer.") This is what o1-pro yielded: https://gist.github.com/carbocation/8d780ad4c3312693ca9a43c61aa97e75 https://gist.github.com/carbocation/8d780ad4c3312693ca9a43c6...
- accrual 2y agoThe Gemini result is quite impressive, thanks for sharing these!
- codedokode 2y agoThey probably trained it for this specific task (generating SVG images), right?
- simonw 2y agoI'm hoping that nobody has deliberately trained on SVG images of pelicans riding bicycles yet.
- pizza 2y agoI do with Claude: https://news.ycombinator.com/item?id=42351796#42355665 https://news.ycombinator.com/item?id=42351796#42355665
- chen_dev 2y agoAmazon Nova models: https://gist.github.com/uschen/38fc65fa7e43f5765a584c6cd24e137f https://gist.github.com/uschen/38fc65fa7e43f5765a584c6cd24e1...