17 ms·
GPT Unicorn has drawn a unicorn
- nullpilot 3y agoFor a second I thought this was about a GPT based product hitting a certain valuation, but I'm much more entertained by the actual content. This is great.
- bgvo 3y agoThought the same thing. And agree.
- nathancahill 3y agoI can't find the Twitter account that was running the "Follow everything GPT says to make $1m" but for a second I thought it was going to be that.
- snordgren 3y agoI'm fairly sure that guy took off with $7,000 from his "investors" and disappeared.
- nathancahill 3y agoI mean, if that's what ChatGPT says to do, who am I to argue?
- bombcar 3y agoTime to release CheatGPT! You can send me seed money and I’ll run off with it, shortening the cycle.
- mkeeter 3y agoHere's a pretty good summary: "What Happened with HustleGPT?" https://thehustle.co/04172023-what-happened-with-hustlegpt/ https://thehustle.co/04172023-what-happened-with-hustlegpt/
- amelius 3y agoFor a second I thought this was about a GPT based agent building a company that turned into a Unicorn.
- tibbydudeza 3y agoI have a sneaky suspicion there is an if-then situation here - always the edge cases :).
- imdsm 3y agoNo sneakiness. Fully open my friend. This is just a bit of fun.
- keepamovin 3y agoThat's actually a pretty epic unicorn. I like how its tail looks like a lightning bolt. Should be a logo idk
- wheelerof4te 3y agoUnikachu.
- matthewfelgate 3y agoValue it a $1Bn to make it a Unicorn Unicorn.
- soxocx 3y agoI would argue that `image-2023-04-25` looks comparable close to a unicorn. The art style does vary a lot though.
- jackdh 3y agoTo me that strikes me slightly more as a bull. It would be interesting to put that in front of someone who isn't precomposed to be looking for a unicorn and see what they say.
- dspillett 3y agoOne or two others, 06-20 for instance, could be a very stylised unicorn too. Though that might be confirmation bias in my human brain making it see patterns it wouldn't have had it not been primed by the word unicorn before looking.
- skinner927 3y agoimage-2023-07-11 has got the spirit
- suby 3y agoI'm confused as to why this would see any improvement over time. Looking at the code, it's by default hitting the gpt 3.5-turbo API. Maybe I'm misremembering, but I thought I've seen statements from people working at OpenAI where it's been claimed that the API is static, we'd be informed of any changes to the underlying model. Is the model actually receiving updates? edit: Looking at previous days, too, it doesn't exactly seem to be improving. I think we just got a lucky sampling.
- tellarin 3y agoYes, the models are updated officially around every three months, with a notice you can still use the previous version for a time until it is decommissioned. Some people claim there are also unannounced changes, but I can't vouch for that. The daily variation is likely due to temperature. To make the response less repetitive.
- calderknight 3y agoThe site linked in the OP is interesting because it takes a picture from GPT every day, so we can see for ourselves if there is any difference with time. We can come back tomorrow and see what it has produced. If it has produces random squigly lines again, we might assume that today's success was just a fluke.
- andai 3y agoWasn't there a study recently that tracked the performance of GPT over time and found significant drop in quality? Did those drops occur at official model changes, or at other times? (i.e. unannounced changes for safety or cost reduction) I mean, if I was OpenAI, I probably wouldn't make an announcement like "we've just quantized the model and increased our profit margins significantly! The only change on your end will be a slightly dumber model. (Don't worry! Most users won't even notice!)"
- n2d4 3y agoThis one [1]? That tracks two distinct versions (0613 vs. 0314). Also, IMO, the tasks they evaluate aren't useful (I rarely want my LLM to tell me whether 17077 is a prime number), and there's room for cherrypicking/survivorship bias. My guess is that OpenAI did something between 0314 and 0613 that shifted focus away from maths to other subjects. [1] https://arxiv.org/pdf/2307.09009.pdf https://arxiv.org/pdf/2307.09009.pdf
- xnorswap 3y agoHow do they get GPT-4 to produce valid SVGs so well? I experimented with SVG generation and it would often produce junk that wasn't even valid as an SVG, and even when it did produce a valid SVG, was often a couple of blobs which it would describe as if it were the mona lisa while just being a couple of elements.
- dannyw 3y agoThis part: https://github.com/adamkdean/gpt-unicorn/blob/master/src/lib/generator.js#L45 https://github.com/adamkdean/gpt-unicorn/blob/master/src/lib...
- xnorswap 3y agoThat'll do it! I wonder how many retries it averages, the images list "tokens", but it's a bit cryptic to work out how to translate that to attempts?
- quickthrower2 3y agoYou can also get the token probs and only sample from the ones that would be valid for each token.
- jwestbury 3y agoLooks like it just reprompts for a valid SVG if the result isn't valid: https://github.com/adamkdean/gpt-unicorn/blob/master/src/lib/generator.js#L45-L50 https://github.com/adamkdean/gpt-unicorn/blob/master/src/lib...
- imdsm 3y agoPretty much. Just checked the logs, that's only happened 3 times since last restart which was 4 weeks ago.
- sigmoid10 3y agoIt has become common knowledge that GPT4 (and also 3.5) have problems with deterministic outputs (even at T=0). So what we're seeing here is just the effect of random sampling, not any actual change to the model itself. If you scroll down, you'll see other close attempts by the exact same model that could already be counted as a win depending on who you ask. Edit: This comment section is a super fascinating case study on the inherent flaws in human cognition. Especially when it comes to seeing patterns in random noise. The fact that some people believe that the model really has to have changed in the past few days is amazing, because if you've kept up with the GPT architecture and the way OpenAI does things (especially on the API), it is incredibly obvious that nothing has happened. But people who want to believe that something has happened will definitely also start to see something.
- bhouston 3y ago100%. This person is trying to find patterns in random noise and believes they are meaningful. The original post hurts my head with its bad logic.
- calderknight 3y agoI don't see the bad logic.
- bhouston 3y agoHe said he is "Asking GPT-4 to draw a unicorn every day to track changes in the model." The variance he is seeing in the output is primarily the product of random chance, rather than changes in the model. Specifically this "unicorn" that he found today is likely just random chance and there was no changes in the model between yesterday and today that lead to it arising. If he wanted to track changes in the model for real, he would have to ask multiple questions per day and try to infer some type of distribution characterization and then see if that changes over time. That is much more complex and not what he is doing. This is just a curious experiment that doesn't mean much.
- 3y ago
- devjab 3y agoIs it just me or doesn't image-2023-04-25 look like a unicorn as well?
- calderknight 3y agoIt looks like an Amogus unicorn
- horse_dung 3y ago4-22 is my favourite… hahaha
- imdsm 3y ago04-22 is how I feel when I post updates to this project and someone tells me it's bad science and I should shut it down
- knolan 3y agoI’ve asked ChatGPT to create models using Blender’s Python API. It typically generates working code. The API has been updated since 2021 so it’s expected that things might break. It mostly just arranges simple spheres and cylinders. You can have it label parts which are usually correct. It struggles with the orientation of things so if you ask it to model a plane the engine nacelles are oriented the wrong way but correctly positioned.
- andrewstuart 3y agoRoom for improvement. Edited for negativity.
- thumbuddy 3y agoIt could help end humanity, with human assistance. For example, expect far more successful phishing campaigns, expect the internet to become a majority of advertising bot garbage, etc.
- Closi 3y agoProbably about as good as I would do if I was hand-coding a SVG and was given an hour. People forget the thing we learnt during the computing revolution - computers don’t need to do individual calculations that humans cannot, they just need to calculate much faster and then that speed can be used to achieve amazing things.
- thumbuddy 3y agoThat's uh a pretty loose interpretation of a unicorn...
- mercurialsolo 3y agoWhat do you think of the current wave of agents spawning off - given that reasoning still needs a lot of grounding? With the probabilistic models at hands, adding reasoning and planning in some sense falls back to traditional software and system design. Do you see any plausible breakthroughs in this area with agent frameworks?
- tellarin 3y agoI'm actually working on this area. I don't see any clear breakthroughs yet, but this is a very exciting area. There is a lot that is not clearly understood about some of the emergent-style abilities of these models or where growth by scaling will hit a limit. So looking into reasoning, what they can and can't do, and what tools/designs, can provide benefits has great potential to inform the next generation of models.
- debarshri 3y agoI can see so many NFTs in this website.
- nathancahill 3y agoI'm sad @ArtDecider has gone dark.
- CrzyLngPwd 3y agoThere's no progress from something that is definitely not a unicorn towards a unicorn, the images seem randomly bad. I think it would be a lot of fun to give it the previous unicorn SVG attempt and ask it to make it more like a unicorn.
- throwuwu 3y agoRead the paper https://arxiv.org/pdf/2303.12712.pdf https://arxiv.org/pdf/2303.12712.pdf Page 7
- gpjanik 3y agoThe prompt must be particularly bad. I managed to get a nicely looking unicorn at the first (and every subsequent) attempt.
- andai 3y agoPls do share!
- fastball 3y agoThe prompts are thus: > system: You are a helpful assistant that generates SVG drawings. You respond only with SVG. You do not respond with text. > user: Draw a unicorn in SVG format. Dimensions: 500x500. Respond ONLY with a single SVG string. Do not respond with conversation or codeblocks. What were yours?
- feitingen 3y agoI usually ask: Give an example of an svg file depicting a whatever. It often gives svg files with incomplete paths, so i tweak the output to be a valid svg file. I also enjoy the conversational description of the drawing. Very often it's well described, ex. "this black circle is the head, and the grey element is the fog" and the drawing is crude like a child's drawing. Landscapes often look better than animals, but animals are sometimes more entertaining.
- gpjanik 3y agoMy prompt was: Imagine you have to draw a SVG of an object. As a model that does not have any idea about how things look, you have to draw "blindly" - as there's no visual feedback, the only feasible tactic is to first list things components each thing consists of (e.g. for a car wheels, windows, chassis, bumpers, lights, etc.) with as much accuracy as you can, establish some constraints (e.g. in a horse legs come out of the body, ears come out of the head, and so on), and then attempt to put all of it in a SVG. This is your task for now, and I will evaluate your drawings. Give me HTML code with embedded SVG that you drew and be verbose about both the things you're going to draw and the constraints. The first thing you will draw is a unicorn.
- TekMol 3y agoRelevant XKCD: https://xkcd.com/904/ https://xkcd.com/904/
- ChatGTP 3y agoUnicow
- pbw 3y agoI made some drawings with GPT-4 Code Interpreter and Pillow. I think with its sub-mage and image composition features you could make some detailed drawings if you were clever about it. [1] https://metastable.org/draw.html https://metastable.org/draw.html
- WolfOliver 3y agolets see how the unicorn looks tomorrow
- streakfix 3y agoPeople keep forgetting that asking GPT to draw is like asking a human to imagine a 6D tesseract.
- capableweb 3y agoOthers keep forgetting that a LLM works nothing like a human, as far as we know today.
- kioleanu 3y agoThis reminds of the artists from 1500s that would do paintings of exotic animals for courts and other rich people, but without ever seeing the actual animal and they would draw them from descriptions.
- itissid 3y agoOne thing people do/did with search engines was to observe their ranking stability over time on common queries like "Cats" or "Dogs". If there was any change one could meaningfully investigate what has changed. Though this seems a bit more like neuropsychological eval by asking a blackbox(AI) questions.
- cvzakharchenko 3y agoI gave the SVG unicorn back to GPT-4 and asked it, what is it. It recognized a head, eyes, body and legs. But it didn't recognize the unicorn. https://chat.openai.com/share/5409b417-b883-429f-893e-abe3d643be6d https://chat.openai.com/share/5409b417-b883-429f-893e-abe3d6...
- flanked-evergl 3y agoYou need to have quite a bit of imagination to say that is a unicorn, but less than you would need for other days.
- danesparza 3y agoThe OP has a slightly different prompt -- so this answer wouldn't qualify. "System: You are a helpful assistant that generates SVG drawings. You respond only with SVG. You do not respond with text. User: Draw a unicorn in SVG format. Dimensions: 500x500. Respond ONLY with a single SVG string. Do not respond with conversation or codeblocks."
- al_be_back 3y agothe output varies wildly ... i'd check again after a few days/weeks and see how it varies. it could be very well be a fluke.
- lewhoo 3y agoWe queried GPT-4 three times, at roughly equal time intervals over the span of a month while the system was being refined, with the prompt “Draw a unicorn in TikZ”. We can see a clear evolution in the sophistication of GPT-4’s drawings Given how random GPT seems with what it's not designed to do the original research is really peculiar. Could it be that they queried GPT on three separate instances some n times and picked the best result ?
- exizt88 3y agoThis is why, as a product manager, you should always test 20 hypotheses per month. At p-value of 0.05 this basically guarantees a successful product feature test every month!
- tantalor 3y agoJust make sure to stop immediately after the trial that validates the hypothesis.
- pxx 3y ago0.95**20 is 0.358; a 64% chance of success is hardly a guarantee
- munchler 3y agoRelevant XKCD: https://xkcd.com/882/ https://xkcd.com/882/
- blackbear_ 3y agoHuh? Either I do not understand what you mean, or you do not understand probability (sorry). My understanding is that you wanted to test whether a certain feature improves user satisfaction or not. Assuming that users have a 0.05=1/20 probability of liking each feature, by testing 20 features you can get at least one successful feature (because 0.05x20=1). This is wrong in two ways. First, the p-value is the probability of observing an effect at least as large as what you measured, assuming that there is actually no effect, i.e., the null hypothesis is true. However, and this is crucial, the p-value does not tell you the probability that the null hypothesis is false (or true)! Again, p-values are unrelated to the truth-ness of the null hypothesis (this is a very common misunderstanding). In fact, if you test 20 hypotheses at a confidence of 5%, you have a probability of (at most) 64% of incorrectly thinking that a feature is useful, while it is not. Second, setting aside hypothesis testing and p-values, if each feature has a 5% probability of being liked and you test 20 features, you only have 64% probability of finding an useful feature. This is because, assuming that the success probabilities of those features are independent and identically distributed, the number of successful features has a binomial distribution [1]. If you wanted to be 95% confident of finding at least one useful feature, you would need to test at least 59 different features each month. What you computed (0.05x20=1) is the expected (average) number of useful features per month over the course of many months. [1] https://en.wikipedia.org/wiki/Binomial_distribution https://en.wikipedia.org/wiki/Binomial_distribution
- sirsinsalot 3y agoIf you get non-deterministic output from a black box, and know that there's both an indeterminate amount of constant random noise in the output AND you do not have any insight into the actual changes being made in the black box... There's nothing to infer with any greater accuracy than the unknown amount of noise in each sample. So, all inference is of unknown accuracy. Also known as useless.
- obblekk 3y agoAn interesting modification would be to have it reflect on its own output each day, and build up a list of advice for future attempts, fed in the next day. That would give it some “learning” and I’d be curious if 1. Would it converge to a consistent shape at all? Or just bounce around random shapes day to day 2. Would it produce unicorns more often than 1/118 times? The hardest part would be getting it to interpret its svg output without seeing it rendered. The multimodal model getting a rendered image would probably be much better, but maybe not!
- imdsm 3y agoDiscussed this with folks last time, but haven't had time to build that version yet. It'd need to be a self-improving unicorn version. If I wasn't going fishing this weekend, I'd build it.
- mnemotronic 3y agoTo me, "image-2023-07-22" looks like Picasso-esque rendering of Charlie Brown (apologies to Charles Schultz).
- jcarrano 3y agoThe methodology is all wrong, as others pointed out. However image-2023-04-26 is pretty interesting. It has some value.
- imdsm 3y agoSorry I got the methodology wrong :image-2023-04-22:
- andsoitis 3y agoAh ask it again a few times today and then again tomorrow, the day after, and the day after that.
- m3kw9 3y agoCould be sheer luck from the randomness of the model, it it came in the 10th draw, you may not have this article
- andsoitis 3y agoIf you presented that image to a random 100 people and asked them what it is, would a substantial number of them say “a unicorn”?
- p1mrx 3y agoCoincidentally, I saw this headline after waking from a dream where a unicorn bashed through the window of my childhood kitchen, and I had to fight it off with a water gun. Melatonin, not even once.
- imdsm 3y agoI've got a blog post for that too! https://adamkdean.co.uk/posts/melatonin-busting-glasses https://adamkdean.co.uk/posts/melatonin-busting-glasses
- danesparza 3y agoWhat is strange to me about this is that with a remarkably similar prompt ("Draw a unicorn in SVG format") I'm able to get this from the Bing image creator (which is powered by DALL-E -- and I would think GPT-4 would have this capability. Perhaps I'm being naive): https://th.bing.com/th/id/OIG.GqpaRZ.NXCN6uxKj7X1u?pid=ImgGn https://th.bing.com/th/id/OIG.GqpaRZ.NXCN6uxKj7X1u?pid=ImgGn
- sp332 3y agoBut it's not in SVG format. It's just a bitmap image that copied the style of a vector image. The Unicorn output from ChatGPT is an actual string of text, that can be rendered into the unicorn image on the page.
- agluszak 3y agoImage generation "in the style of SVG" (which DALL-E did) has nothing to do with generating actual svg source code (which GPT does) and then rendering it.
- lolinder 3y agoWhen you prompt GPT-4 with the API, you can only get text out. In this case OP is asking for text that happens to define an SVG file (which is a text format). When you ask Bing Image Creator to produce an image, your prompts goes to the image model and an image comes out. It's a bitmap image, not an SVG—that unicorn is defined not as text but as a series of pixels with different colors. Comparing the two is comparing apples and oranges, because there are completely different models underneath and completely different output formats.
- oneearedrabbit 3y agoNot directly related to the post, but still feels somewhat relevant. Back in March I used a bit more elaborate multi-step prompts for GPT3.5 to generate amusing pictures and published a gallery [1]. However, I eventually reached a point where changing prompts did not consistently improve the final results. At the end of the day, the quality of images are only as good as the training dataset, and GPT is a black box. For something different, to test whether it is possible to "compress" visual content specifically for GPT, I ran another experiment. SVG, being a verbose format, takes time to generate a detailed image, and it also becomes expensive over time. I translated a subset of SVG elements into Forth words [2], which has a nice synergy with GPT tokens--this allowed me to progressively render pictures and produce smaller outputs without sacrificing much in quality. Finally, I training my own GPT2-like model on the QuickDraw dataset [3]. It's not surprising that a sequence transformer can be trained to produce coherent brush strokes and recognizable images as long as there is a way to translate a graphical content into a sequence of tokens. That said, I found myself with more questions than I started, and trying other ideas now. [1] https://drawmeasheep.net/pages/about.html https://drawmeasheep.net/pages/about.html [2] https://drawmeasheep.net/pages/gpt-forth.html https://drawmeasheep.net/pages/gpt-forth.html [3] https://drawmeasheep.net/pages/nn-training.html https://drawmeasheep.net/pages/nn-training.html
- mmh0000 3y agoI was curious about how they were getting GPT to generate images, the prompting is so simple[1] that GPT still blows my mind: async fetchImage(context) { const messages = context || [ { role: 'system', content: `You are a helpful assistant that generates SVG drawings. You respond only with SVG. You do not respond with text.` }, { role: 'user', content: `Draw a unicorn in SVG format. Dimensions: 500x500. Respond ONLY with a single SVG string. Do not respond with conversation or codeblocks.` } ] const response = await this.api.generateCompletion(messages) if (!isSvg(response.content)) { console.error('Generated image is not valid SVG:', response.content) messages.push({ role: 'assistant', content: response.content }) messages.push({ role: 'user', content: `The generated image is not valid SVG. Please try again. Only respond with SVG code. No text.` }) return this.fetchImage(messages) } console.log('Generated image:', response.content) return response } [1] https://github.com/adamkdean/gpt-unicorn/blob/master/src/lib/generator.js https://github.com/adamkdean/gpt-unicorn/blob/master/src/lib...
- melicerte 3y agoDoes it look like a unicorn to you ? It looks more like a cat with a spaghetti on his head... image-2023-04-25 looks more like an unicorn to me (although more a cow based unicorn than a horse based unicorn). Which leads me to this genuine question: : how close do the images resemble a unicorn? I mean how can one track resemblance and how to draw a line saying GPT has drawn an unicorn?
- xtracto 3y agoSeveral months ago when ChatGPT was just published, I asked it to draw a diagram of the contents of a cell (nucleus, membrane, amynoacids etc) in SVG (similar to the experiment shown here). At that time, it generated an OK SVG drawing, with very basic figures (squares, trianlges, circles, etc) representing the elements in a cell. I did this test to show my Biologist mom, the "power" of this new AI technology. Fast forward a couple of months later, I tried it again, and it was "blocked": It kept telling me "as an AI I cannot draw", even after emphazising the part of "generate SVG code". For me, it was another example of OpenAI "borking" the capabilities of ChatGPT.
- Spivak 3y agoUse the playground and it will work a lot better. I gave it about 500 tokens and it got to the end. > Write SVG code to make a basic diagram of a cell showing structures such as the nucleus, ribosomes, and mitochondria. Here's a very basic example of SVG code to represent a cell that includes structures like the nucleus, ribosomes, and mitochondria: ```svg <svg width="200" height="200" xmlns="http://www.w3.org/2000/svg"> <!-- The entire cell --> <circle cx="100" cy="100" r="100" style="fill:#FF9999;" /> ... ```
- aaroninsf 3y agoHas anyone set up a cafe press site that automatically slaps each day's image on a T-shirt? The most recent one would be good.
- esafak 3y agoThe prompt is Draw a unicorn in SVG format. Dimensions: 500x500. Respond ONLY with a single SVG string. Do not respond with conversation or codeblocks. https://github.com/adamkdean/gpt-unicorn/blob/master/src/lib/generator.js#L40 https://github.com/adamkdean/gpt-unicorn/blob/master/src/lib... I just asked this and got the following result: <svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" width="500" height="500" viewBox="0 0 500 500"> <path fill="#FDD7E4" d="M244.898 309.347c-.355-.421-5.388-6.485-5.388-10.305v-40.614l-3.073-12.759c-.708-2.684-2.527-4.924-5.034-6.38l-24.773-14.334c-8.152-4.72-15.709-10.72-17.905-18.788l-12.3-35.006c-.845-2.696-3.577-4.61-6.55-4.61h-48.235c-2.973 0-5.705 1.914-6.55 4.61l-12.3 35.005c-2.196 8.068-9.753 14.067-17.905 18.787L92.998 235.968c-2.507 1.456-4.325 3.696-5.034 6.38l-3.073 12.759v40.615c0 3.82-5.033 9.884-5.388 10.305-9.168 10.836-14.727 24.606-14.727 39.49v42.812C39.373 426.93 69.062 456.62 105.34 456.62h17.215c1.256 21.698 20.249 38.969 42.405 38.969s41.15-17.271 42.405-38.969h17.48c1.256 21.698 20.25 38.969 42.406 38.969s41.15-17.271 42.404-38.969H394.66c36.278 0 65.968-29.69 65.969-65.969v-42.812c0-14.884-5.559-28.654-14.727-39.491zm-189.9-4.337a77.079 77.079 0 0 0-4.225-.302c-19.322 0-35.012 15.689-35.012 35.012V375.99h74.25v-66.968c0-19.323-15.69-35.012-35.013-35.012zm208.12 0c-19.323 0-35.012 15.689-35.012 35.012v66.968h74.25v-66.968c0-19.323-15.69-35.012-35.012-35.012zm-118.63-66.968c10.286 0 18.632 8.345 18.632 18.632v48.424h-37.264v-48.424c0-10.287 8.345-18.632 18.632-18.632zm-27.944-18.633c-19.323 0-35.012 15.689-35.012 35.012V341.98H351.6V275.27c0-19.323-15.69-35.012-35.012-35.012zm-35.013-17.911c-10.287 0-18.632-8.346-18.632-18.633s8.345-18.632 18.632-18.632 18.632 8.346 18.632 18.632-8.345 18.633-18.632 18.633z"/></svg> I am not going to fault a language model for not getting that right! This is fundamentally not a language task. It demands an image model.
- fl7305 3y agoI would imagine that it would do better if it was told to build it up step by step, and critique its own work as it goes along.
- imdsm 3y agoIt would but what I like is how difficult the task is. If the model becomes able (like today) to generate passable results from such a simple prompt, then that shows something, IMO.
- cushpush 3y agothe fact that some of the early ones are anti-smiles, a playful takeaway could be that it saw the insurmountability of the challenge at first!