7 ms·
So during my Nano Banana Pro experiments I wrote a very fun prompt that tests the ability for these image generation models to follow heuristics, but still requ
by minimaxir 6mo ago
So during my Nano Banana Pro experiments I wrote a very fun prompt that tests the ability for these image generation models to follow heuristics, but still requires domain knowledge and/or use of the search tool:
Create a 8x8 contiguous grid of the Pokémon whose National Pokédex numbers correspond to the first 64 prime numbers. Include a black border between the subimages.
You MUST obey ALL the FOLLOWING rules for these subimages:
- Add a label anchored to the top left corner of the subimage with the Pokémon's National Pokédex number.
- NEVER include a `#` in the label
- This text is left-justified, white color, and Menlo font typeface
- The label fill color is black
- If the Pokémon's National Pokédex number is 1 digit, display the Pokémon in a 8-bit style
- If the Pokémon's National Pokédex number is 2 digits, display the Pokémon in a charcoal drawing style
- If the Pokémon's National Pokédex number is 3 digits, display the Pokémon in a Ukiyo-e style
The NBP result is here, which got the numbers, corresponding Pokemon, and styles correct, with the main point of contention being that the style application is lazy and that the images may be plagiarized: https://cdn.bsky.app/img/feed_fullsize/plain/did:plc:oxaernim5mj2mmy3ytrvb42n/bafkreigussguihcycvzm3aaosw4gzvgwhibwmjiknkkatfcaeajapmrqca https://cdn.bsky.app/img/feed_fullsize/plain/did:plc:oxaerni...
Running that same prompt through gpt-2-image high gave an...interesting contrast: https://cdn.bsky.app/img/feed_fullsize/plain/did:plc:oxaernim5mj2mmy3ytrvb42n/bafkreidgbmtivl33ohwgbvylnvqzo2uz43zkuwu65cg42wkk7b7up7emje https://cdn.bsky.app/img/feed_fullsize/plain/did:plc:oxaerni...
It did more inventive styles for the images that appear to be original, but:
- The style logic is by row, not raw numbers and are therefore wrong
- Several of the Pokemon are flat-out wrong
- Number font is wrong
- Bottom isn't square for some reason
Odd results.
- rrr_oh_man 6mo agoWhy would you consider this a good prompt?
- minimaxir 6mo agoBecause both Nano Banana Pro and ChatGPT Images 2.0 have touted strong reasoning capabilities, and this particular prompt has more objective, easy-to-validate criteria as opposed to the subjective nature of images. I have more subjective prompts to test reasoning but they're your-mileage-may-vary (however, gpt-2-image has surprisingly been doing much better on more objective criteria in my test cases)
- o10449366 6mo ago[flagged]
- minimaxir 6mo ago"Quirky and obscure" has the functional benefit of ensuring the source question is not in the training data/outside the median user prompt, and therefore making the model less likely to cheat. We have enough people complaining about Simon Willison's pelican test.
- o10449366 6mo agoWhen you program, do you consider using your prior knowledge of programming cheating?
- codemog 6mo agoAh yes, also known as C++ enjoyers.
- tailscaler2026 6mo agostill #opentowork huh
- beepbooptheory 6mo agoWhere does one even use that hashtag?
- minimaxir 6mo agoIt's a LinkedIn joke.
- Bjartr 6mo agoWhat would make the prompt a better actual evaluation in your judgement?
- leptons 6mo ago
- dvt 6mo agoThis is an amazing test and it's kinda' funny how terrible gpt-2-image is. I'd take "plagiarized" images (e.g. Google search & copy-paste) any day over how awful the OpenAI result is. Doesn't even seem like they have a sanity checker/post-processing "did I follow the instructions correctly?" step, because the digit-style constraint violation should be easily caught. It's also expensive as shit to just get an image that's essentially unusable.
- anshumankmr 6mo agothat is interesting cause I feel gpt-image-1 did have that feature. (source: https://chatgpt.com/share/69e83569-b334-8320-9fbf-01404d18dfa5 https://chatgpt.com/share/69e83569-b334-8320-9fbf-01404d18df...)
- weird-eye-issue 6mo agoYou are comparing ChatGPT to a raw image model. These are two completely different things. ChatGPT takes your input, modifies the prompt and then passes it to the image model and then will maybe read the image and provide output. The image model like through the API just takes the prompt verbatim and generates an image.
- minimaxir 6mo agoNano Banana Pro and ChatGPT Images 2.0 also tweak the prompt because they can think.
- weird-eye-issue 6mo agoYes exactly, "ChatGPT Images 2.0" is in ChatGPT. That is not a model.
- hyperadvanced 6mo agoI wouldn’t say it’s terrible. I wouldn’t say it’s a huge step forward in terms of quality compared to what I’ve seen before from AI
- heroku 6mo ago[dead]
- Razengan 6mo agoEven a few months ago, ChatGPT/Sora's image generation performed better than Gemini/Nano Banana for certain weird prompts: Try things like: "A white capybara with black spots, on a tricycle, with 7 tentacles instead of legs, each tentacle is a different color of the rainbow" (paraphrased, not the literal exact prompt I used) Gemini just globbed a whole mass of tentacles without any regards to the count
- m3kw9 6mo agoProb a very unscientific way to test an image model. This would me likely because they have the reasoning turned down and let its instant output takeover
- minimaxir 6mo agoThere's no good scientific way to test a closed-source model with both nondeterministic and subjective output. This example image was generated using the API on high, not the low reasoning version. (it is slow and takes 2 minutes lol)
- crustaceansoup 6mo agoIf the results are quantifiable/objective and repeatable it's scientific, how is it not scientific? The reasoning amount is part of the evaluation isn't it?
- TeMPOraL 6mo agoThis is the best kind of science there is: direct, empirical test.
- vincentbuilds 6mo agobanana Pro gets the logic and punts on the art; gpt-2-image gets the art and punts on the logic. Feels like instruction-following and creativity sit on opposite ends of the same slider.
- dieortin 6mo agoThis feels incredibly AI generated
- doginasuit 6mo agoThe random accusations of AI generated comments are the most annoying part of the unfolding AI dystopia.
- fennecfoxy 6mo agoDogs don't wear suits! You must be AI too ;)
- doginasuit 6mo agoDamn, busted!
- sdwr 6mo agoYeah it's fascinating to have an alternate source for intelligence, feels like a mental Rosetta Stone
- changyou 6mo ago[dead]
- MrManatee 6mo agoPrompts like this feel like it's using the wrong abstraction. The "obvious" thing to do with something like this would be to generate some code that generates the image and then run that code. Inspired by this, I tried something much simpler. I asked it to draw 12 concentric circles. With three tries it always drew 10 instead. https://chatgpt.com/share/69e87d08-5a14-83eb-9a3b-3a8eb1469237 https://chatgpt.com/share/69e87d08-5a14-83eb-9a3b-3a8eb14692...
- LeifCarrotson 6mo agoI think prompts like this are where agentic workflows come in to play. If you asked it to do generate the first 64 prime numbers, AI tools could do that. If you asked it to draw a charcoal image of Pokemon 13, it could do that. If you asked it to add a white Menlo 13 on a black background to the top left corner of that image, it could do that. If you asked it to do that 63 more times, it could do those things, and if you asked it to assemble those into a grid, it could. It can't get that in a one-shot. Perhaps, though, it could figure out when it needs to break a problem into individual tasks to delegate to itself and assemble them at the end.
- wahnfrieden 6mo agoThat's what makes it a fair evaluation of its limits
- fennecfoxy 6mo agoI mean asking these transformers to do maths has always been the wrong task. It's like we're now considering "it doesn't have x tools built with traditional code built in". Though I suppose we're testing their model + agent harness here as well. It really _should_ have all of those tools/reasoning available to accomplish a task like the above without issue.
- wahnfrieden 6mo agoIt's only been the wrong task because they've been deficient at it and expensive to use, so we had workarounds. They are getting better at these tasks and cheaper (sometimes). It's fair to evaluate even if there are more economical and accurate alternatives available.
- razorbeamz 6mo agoNeither of them drew them in an 8-bit style either. It's way too many colors.
- dodslaser 6mo agoMaybe they're so advanced they learned to write to the palette registers mid-scanline.
- podgietaru 6mo agoThey made the same mistake a lot of people do, 8-bit meaning Retro style. But they're from the 16bit(?) GBA games.
- AussieWog93 6mo agoFor what it's worth, NBP made some mistakes too. Artistic oddities aside (why are the 8-bit sprites 16-bit, why do the charcoal drawings have colour, why does the art of specifically the Gen 1 Pokemon look so off.), 271 is Lombre, not Lotad.
- pfortuny 6mo agoJust try a 23-sided plane convex polygon.
- Palmik 6mo agoI do not think this is a good prompt or useful benchmark, but nonetheless, it seems to work better for me: https://chatgpt.com/share/69e88a94-ded8-8395-b5dc-abceb2f44d02 https://chatgpt.com/share/69e88a94-ded8-8395-b5dc-abceb2f44d...
- minimaxir 6mo agoHuh, that is indeed better. If ChatGPT Images 2.0/gpt-2-image is more nondeterministic than usual, than that is in itself a useful data point.
- podgietaru 6mo agoHow is it that a model can produce what must be near 1:1 images ripped straight out of Pokemon Fire Red (The first ones) for profit and not be infringing copyright. I know that's the game, but it seems CRAZY to me that they can do this.
- Jensson 6mo agoGemini uses google search to find references when making images, so it probably found the pokemon images online to do this. > I know that's the game, but it seems CRAZY to me that they can do this. Its not crazy that a search can find existing pokemon images. Maybe google should show which images it used as references to be more transparent here.
- dd8601fn 6mo agoThe funny this is the main complaint I’ve heard so far is that it repeatedly refused to operate on original content… because it might violate copyright.
- DrewADesign 6mo agoYeah, the CSAM generated by grok proves the guardrails are only really good for stymieing benign uses.
- minimaxir 6mo agoIn testing the ChatGPT interface it appears to be looser on copyright than expected given the legal trouble.
- DrewADesign 6mo agoIt can’t. It violates copyright. The big players are the only ones with the money to pursue these things, but they’re interested in replacing artists with AI trained on their models so they settle and set up some sort of agreement. The little guys have no presidential case law to help them along, and nowhere close to the resources to push it that far, so they get steamrolled. I know artists famous enough for people— even commercial entities — to regularly blatantly rip them off by name with “in the style of” prompts, but there’s no realistic path to pursue it. Fame doesn’t pay legal bills.
- Melatonic 6mo agoBut where is the one for Missing No ?!