3 ms·

Hey everyone! We made this as a hackathon project over the weekend to experiment with OpenAI’s new CLIP model. https://openai.com/blog/clip/ https://openai.com/

Hey everyone! We made this as a hackathon project over the weekend to experiment with OpenAI’s new CLIP model. https://openai.com/blog/clip/ https://openai.com/blog/clip/ Glad you’re enjoying playing with it and discovering failure modes! Yesterday we tweaked the prompts to penalize people writing text when we discovered CLIP can read. Looks like we should add something to prevent it from falling in love with solid blocks of color. Any ideas for how to describe that? (Someone below suggested a bonus for “complexity” but I’m concerned that’d make it fall in love with scribbles.) You can read more about the project on my Show HN from yesterday: https://news.ycombinator.com/item?id=25822610 https://news.ycombinator.com/item?id=25822610

yeldarb 6y ago
Update: "An illustration of an empty canvas filled with color" seems to work pretty well to extract the solid blocks of colors from the actual drawings. We're adding that to our prompts now and then will recalculate the leaderboards. Please surface any other failure modes you find! It's pretty neat to learn about what trips it up; it's brand new so you all are amongst the first to poke at it.
- yeldarb 6y ago
  Update 2: appears this was actually a bug in my code that applies a text penalty :blush: These blocks of color weren't scoring highly for the prompt... they were scoring really low for similarity to text... which ended up giving them a sort of anti-text bonus relative to other submissions. The fix should be up shortly. Update 3: leaderboard should be mostly fixed! Update 4: leaderboard fix fully implemented. And they're looking good! There are some artists amongst you. https://paint.wtf/leaderboard https://paint.wtf/leaderboard
prionassembly 6y ago
A bonus for complexity can be given according to something like this: https://ai.stanford.edu/blog/topologylayer/ https://ai.stanford.edu/blog/topologylayer/
soulofmischief 6y ago
I drew a circle and it said "Potentially NSFW". Neat idea, thanks for sharing.
- yeldarb 6y ago
  You can click to reveal those if you want. It runs each image through a variety of prompts including screens for racist symbols, nsfw drawings, and nudity. If you send me a link to yours I can let you know what it tripped up. (One of our heuristics is "is it closer to nudity than to the prompt" so if it wasn't a good representation of the prompt it could be more that it was far away from the target vs being particularly close to the nsfw target.)

yeldarb 6y ago

Update: "An illustration of an empty canvas filled with color" seems to work pretty well to extract the solid blocks of colors from the actual drawings. We're adding that to our prompts now and then will recalculate the leaderboards. Please surface any other failure modes you find! It's pretty neat to learn about what trips it up; it's brand new so you all are amongst the first to poke at it.

yeldarb 6y ago
Update 2: appears this was actually a bug in my code that applies a text penalty :blush: These blocks of color weren't scoring highly for the prompt... they were scoring really low for similarity to text... which ended up giving them a sort of anti-text bonus relative to other submissions. The fix should be up shortly. Update 3: leaderboard should be mostly fixed! Update 4: leaderboard fix fully implemented. And they're looking good! There are some artists amongst you. https://paint.wtf/leaderboard https://paint.wtf/leaderboard

prionassembly 6y ago

A bonus for complexity can be given according to something like this: https://ai.stanford.edu/blog/topologylayer/ https://ai.stanford.edu/blog/topologylayer/

soulofmischief 6y ago

I drew a circle and it said "Potentially NSFW". Neat idea, thanks for sharing.

yeldarb 6y ago
You can click to reveal those if you want. It runs each image through a variety of prompts including screens for racist symbols, nsfw drawings, and nudity. If you send me a link to yours I can let you know what it tripped up. (One of our heuristics is "is it closer to nudity than to the prompt" so if it wasn't a good representation of the prompt it could be more that it was far away from the target vs being particularly close to the nsfw target.)