6 ms·
Now add a walrus: Prompt engineering in DALL-E 3
- tim-fan 3y agoAny ideas on if/when (image+text)->image models will be available? Would be good to be able to iterate on images (keep this, change that etc). The use of the seed looks useful but I'm guessing it has its own limitations.
- qingcharles 3y agoDALL-E 2 and now DALL-E 3 have given me more laughs than anything in years. There used to be a video game magazine which would rate games by "Improvements through improper play." That's exactly how I feel about DALL-E. There are several Subreddits and Facebook Groups I've submitted some seriously cursed AI output to. GPT-V is a total marvel too. I just used one of the medieval Chaucer images from the recent HN post about their digitization, and told GPT my wife had left me a funny note this morning that I needed to read. It transcribed and translated it perfectly, even though it was practically unreadable.
- blorenz 3y agoSimon Willison came into my consciousness so many, many years ago because of Django. Though Django is an integral part of my daily jam, now I'm here for the LLM work. Please keep it up! I wish I had an immediate use of Datasette but I'll undoubtedly get to that one day. Thank you for what you do.
- vunderba 3y agoThe fact that you get access to DALL-E 3 "for free" if you're already subscribed to ChatGPT Pro is going to give MJ and other competitors a serious run for their money. Also being able to reuse a seed to emulate a InstructPix2Pix architecture is a game changer.
- trompetenaccoun 3y agoMaybe I'm not their typical target demographic but OpenAI's products are completely useless to me, the way they've neutered them and are restricting the output. For images I'd rather run Stable Diffusion locally. You own what you produce with it too (with some caveats). GPT-3 was cool before they came out with the chat version but it's been all downhill from there.
- willsmith72 3y agoI don't find the the quality bad at all, and find them extremely useful. So far it's my daily assistant for software architecture, business plans, marketing content, debugging, summarisations, and many more. Have you used gpt4?
- Spivak 3y agoThe filter categories for OpenAI's moderation API are hate, harassment, self-harm, sexual, sexual/minors, and violence. Is it really the end of the world that DALL-E is rated PG-13? There's no limit to the available models to generate adult content, but having one that doesn't makes it embedable in other applications.
- garblegarble 3y ago>You own what you produce with it too As I understand it, this is incorrect - all of these outputs are the creations of a machine, and as such are not eligible for copyright protection
- DharmaPolice 3y agoSurely a manual modification of the image would be a fairly easy way around this? Anyway, that may not matter - I think what people are looking for here is the absence of being sued by someone else, not the ability to sue someone else. If I generate an image to put in my companies annual report, I'm not worried about someone else copying the image somewhere else.
- SamBam 3y agoThere was a widely-misunderstood lawsuit recently that lots of people interpreted this way, but it wasn't what the lawsuit concluded at all. "Creations of a machine" is meaningless. Machines are tools we use to be creative, even if we're getting some things out that we do not expect. When a 3D program renders light into a scene, does that mean the image is not copyrightable?
- objectivetruth 3y ago
- ranit 3y agoWhere on ChatGPT console can you access DALL-E 3? I am ChatGPT Pro subscriber, but all I can access, additional to GPT-4, are beta features (plugins and Advanced data analysis).
- SamBam 3y agoI also don't seem to have access. It seems that it's being slowly rolled out to subscribers, and maybe you and I are on the tail end.
- ranit 3y agoI just got it!
- Auracle 3y agoFor me it’s available in the app but not the website, although I can access the conversations that have generated images on the website.
- GaggiX 3y agoYou can use Dalle 3 completely free on the Bing AI website.
- qingdao99 3y agoAnd it's far better in my opinion - ChatGPT actually creates its own prompts from what you give it and feeds those to DALL-E (you can see this by clicking on the images it returns and reading what the actual prompt was), although you can disable this by just firmly telling ChatGPT not to modify your prompt. Additionally, the image generation itself is quite different. ChatGPT's DALL-E seems to create much more stylized images - much harder to get plain shots that don't heavily embellish your description.
- TeMPOraL 3y agoOpenAI justified that in a paper the other day, saying that DALL-E 3 performs better on longer, more detailed prompts describing all aspects of the image in rich language - so they put GPT-4 in front to expand the typical user's short and vague prompt, so such users get nicer results by default. My own observation: this kind of hack is possible only since/with GPT-4 - it takes an LLM this powerful to reliably extend and enrich arbitrary user input into much longer prompt, that's coherent, consistent, and a plausible (to human) interpretation of the original input. Now this may fan the flames on the "is it or is it not" AI discussions, but: you could almost say that GPT-4 is engaging in creative process here.
- msikora 3y agoIs DALL-E 3 and GPT-V now supported in the API?
- vunderba 3y agoThere is no GPT-V, there is GPT-4V which adds the image recognition component to GPT-4. I believe the API for both this and DALL-E 3 is releasing sometime early November.
- Kiro 3y agoWhere did you see "early November? I just spent a lot of time looking for a date and could only find the official blog post being vague about it: > DALL·E 3 is now in research preview, and will be available to ChatGPT Plus and Enterprise customers in October, via the API and in Labs later this fall.
- doctorpangloss 3y agoSure Simon. Let's see you try to get a minimalist image out of it.
- BoorishBears 3y agoDo these count? https://imgur.com/a/Tec7RbY https://imgur.com/a/Tec7RbY
- keyle 3y agoLove the idea of prompting for sesame street characters, someone tell him his F1 cars are on 3 wheels in the last image though...
- althea_tx 3y agoAre you sure the seed number isn’t a hallucination? I think it might be an internal reference which vaguely alludes to the original prompt. The seed has no persistent meaning beyond the chat instance, so I think you could get same effect by referring a previous image with prose.
- willsmith72 3y agoI thought the same, it seems strange it would be added to that json as a key separate to the prompt though
- Filligree 3y agoWhy? That's just how the image generators technically work.
- willsmith72 3y agoBecause it seems to me like poor engineering to leak it. Not disputing it, just surprised to see it
- janalsncm 3y agoMaybe I’m missing something. What is the threat of exposing the seed? It just makes images reproducible.
- TeMPOraL 3y agoRemember that just like with every other model they expose, DALL-E 3 API is stateless. GPT-4 to DALL-E 3 hand-off is implemented client side (similar to plugins), so the entire exchange goes through your machine. Everything - seed included - starts "leaked" by design.
- simonw 3y agoThe seed isn't a hallucination, I proved that to myself by reviewing the underlying JSON, see this section: https://simonwillison.net/2023/Oct/26/add-a-walrus/#peeking-under-the-hood https://simonwillison.net/2023/Oct/26/add-a-walrus/#peeking-... { "prompts": [ "Photo of two Muppet characters: a pelican with a monocle and a bow tie, and a walrus with big, goofy tusks and a dapper bow tie. They're seated in a Muppet-style commentary booth, providing humorous commentary on the Monaco Grand Prix. Cartoonish F1 cars race by, and colorful yachts are seen in the distance." ], "size": "1024x1024", "seeds": [1379049893] }
- ChatGTP 3y agoI love the way the cars are often “driving on water”. So perfect yet so wrong.
- nicholassmith 3y agoSimulating Red Bulls next set of upgrades no doubt.
- jallmann 3y agoVery interesting that ChatGPT seems to prompt Dall-E via the client, rather than keeping that interaction entirely server-side. Keeping it server-side would be less likely to leak details as seen here, and makes it less susceptible to tampering. Also nice to see that Dall-E 3 seeds were finally fixed. That must have happened within the last week or so; they weren't working last I checked (chatgpt always used a fixed seed of 5000). > Midjourney and Stable Diffusion both have a “seed” concept, but as far as I know they don’t have anything like this capability to maintain consistency between images given the same seed and a slightly altered prompt. I suspect this is more a function of Midjourney's prompt adherence being fairly poor right now. Even so, the images often aren't dramatically different. Example: https://analyzer.transfix.ai/?db=josh&q=%28robot+%7C%7C+android%29+%26%26+governess+%26%26+sargent+%26%26+raw+%26%26+--seed+3358341895+%26%26+--s+100+%26%26+--ar+9%3A16+%2F%2F+display+4+col+%2F%2F+label+%24prompt+%23+%24idx+%2F%2F+sort+prompt+%2F%2F+layout+grid+%2F%2F+imgHeight+45 https://analyzer.transfix.ai/?db=josh&q=%28robot+%7C%7C+andr...
- Zetobal 3y agoThe ip2adapter from Tencent does the same for SD.
- danShumway 3y ago> Very interesting that ChatGPT seems to prompt Dall-E via the client, rather than keeping that interaction entirely server-side. Keeping it server-side would be less likely to leak details as seen here, and makes it less susceptible to tampering. I don't have access to test, but given OpenAI's record on stuff like this, it would be a good idea for someone to check to see whether users can resend/intercept those requests to directly control the prompts that are sent to Dall-E without going through GPT. Most likely they're only part of conversation history and they're unmodifiable, but I wouldn't necessarily take it as a given, and it would be quick for anyone with access who knows their way around the browser dev tools to check.
- teaearlgraycold 3y agoYup you can control it directly easily enough: Generating images in one chat: https://i.imgur.com/sIKSfCy.png https://i.imgur.com/sIKSfCy.png Reproducing exactly in another: https://i.imgur.com/C8Tqo48.png https://i.imgur.com/C8Tqo48.png
- yreg 3y ago> People have been trying to figure out hacks to get Midjourney to create consistent characters for the past year, and DALL-E 3 apparently has that ability as an undocumented feature! [by reusing the seed] Using a constant seed to produce similar images has been the technique from the very start but it has limitations. You cannot e.g. keep the character consistant between different poses in this way.
- muskmusk 3y agoThose prompts are wild. It is deeply impressive that it works. Think about what would happen if you gave such instructions to a human. Would they be able comply? How big is the overlap between people who are creative enough to produce the kind of pictures Dall-E produces and disciplined enough to follow complex instructions so rigourously? It also is just straight up impossible to convert those instructions to "regular" code.
- rainmouse 3y agoI can't help but feel that any perceived overlap is coincidental. An illusion similar to seeing a face in an abstract drawing that the artists, or in this case algorithm developers are keen to exploit. Our need to find the fimiliar in something that is ultimately completely alien to our way of thinking. But with enough existing prompts and training data, it will continue to learn and better trick our senses. I totally agree that putting those instructions into code would be outrageously complicated, and the biggest strength here is it's ability to the gist of what we are trying to convey.
- thfuran 3y agoBut it mostly didn't comply. The overlong prompts consistently contain details that are ignored.
- iamflimflam1 3y agoWhat’s interesting to me from this is that the technique they are using to integrate dalle into ChatGPT is pretty much the same as they are using for plugins.
- bertylicious 3y agoAm I the only one appalled by the audacity to call the writing of vague instructions for an AI "engineering"? However many levels of indirection there might be at play.
- jojobas 3y agoIf social engineering counts, this should too.
- bertylicious 3y agoFair point. Maybe the term engineering has a tradition of being used in contexts that are not associated with hard science. Another example: the term "software engineering" was invented to point out the lack of scientific foundation for programming practices and to motivate the community to fix that. Or maybe it's just me associating engineering with hard science.
- smokel 3y agoThis made me look into the etymology of "engineer", and I was slightly surprised that it is all about weapons, and not so much bridges.
- muzani 3y agoI see two definitions of engineering: 1. Building engines from scratch, i.e. something that converts a thing from X to Y consistently. 2. Knowing enough about an engine to get it to do what you want it to do, as well as maintaining it. Prompt engineering initially started from the latter, back before ChatGPT and stuff which you'd just give instructions to. Engineers have to be familiar enough to know how it works, what it's strong at, what it isn't, things like margin of failure, all that stuff. A lot of it is just prompt alchemy, I guess, but the lower part of this article has some juicy reverse engineering.
- intelVISA 3y agoAs opposed to writing vague instructions for a compiler?
- jojobas 3y agoOf course it has to produce "diversity". I wonder what's the incentive to do that.
- smcl 3y ago(edit: wrong)
- Kiro 3y agoThe "second one" he's referring to does not have those issues. https://static.simonwillison.net/static/2023/dalle-3/add-walrus-2.png https://static.simonwillison.net/static/2023/dalle-3/add-wal...
- rainmouse 3y agoGave this a try on chat GPT 4 enterprise, it confirmed it could generate images using a text-to-image model similar to DALL-E, but when prompted to, all it says is.... "I'm unable to generate or provide the image at the moment. However, you can use the description you've provided with an AI image generation tool like DALL-E or a similar service. They can create detailed and imaginative visuals based on textual prompts like yours." Guess I'll try again later.
- pmx 3y agoYou need to make sure you select the correct model to do it. Hover over GPT-4 and select DALL-E 3. Should work then!
- rainmouse 3y agoAh yeah, many thanks for the clarification. I notice that UI drop down. As of 27th October 2023, you are currently much more helpful than chat GPT-4 ;) When asked about this it said it could do this. When Prompted it decided it couldn't. When asked for an explanation it gave me; "I apologize for the confusion and any frustration it may have caused. As of my last training cut-off in April 2023, I am not equipped with the capability to generate images directly within this chat interface. My earlier response was incorrect, and I appreciate your understanding as I correct this mistake."
- k3liutZu 3y agoThank you for this, I had trouble figuring out how to try out Dalle from ChatGPT
- qwertox 3y agoA couple of days ago I didn't notice that ChatGPT was still set to Dall-E, because it was just helping me along "as usual" with my programming tasks without giving any hints of being in this Dall-E-Mode. When I noticed this, I asked it to generate an image of what has been discussed so far, where the first image turned out to be pretty nice [0] We were dealing a lot with timestamps, NumPy, Pandas stuff. https://imgur.com/a/SBZ36KT https://imgur.com/a/SBZ36KT