11 ms·

Exploring 12M of the 2.3B images used to train Stable Diffusion

446 points by detaro 4y ago

rektide 4y ago
So excellent. Flipping the story we see all the time on it's head. AI's quasi-mystical powers are endless spectacle. Taking a look through the other side of the looking glass is vastly overdue. Amazing work. Just just just starting to scratch the surface. 2% of the data gathered, sources identified. These are a couple of sites we can now source as the primary powerers of AI. Barely reviewed, dived into in terms of the content itself. We have so little sense & appreciation for what lurks beneath, but this is a go.
- benreesman 4y ago
  It is an unambiguous social necessity to demystify these things. In a world where the lay public didn’t really know about photoshop, photoshop would be a terrifying weapon. Likewise modern ML is for the most part mysterious and/or menacing because it’s opaque and arcane and mostly controlled by big corporate R&D labs. Get some charismatic science popularizers out there teaching people how it works, all the sudden not such a big scary thing.
lmarcos 4y ago
I always had the crazy idea of "infinite entertainment": somehow we manage to "tap" into the multiverse and are able to watch TV from countless of planets/universes (I think Rick and Morty did something similar). So, in some channel at some time you may be able to see Brad Pitt fighting against Godzilla while the monster is hacking into the pentagon using ssh. Highly improbable, but in the multiverse TV everything is possible. Now I think we don't need the multiverse for that. Give this AI technology a few years and you'll have streaming services a la Netflix where you provide the prompt to create your own movie. What the hell, people will vote "best movie" among the millions submitted by other people. We'll be movie producers like we are nowadays YouTubers. Overabundance of high quality material and so little time to watch them all. Same goes for books, music and everything else that is digital (even software?).
- gfody 4y ago
  > So, in some channel at some time you may be able to see Brad Pitt fighting against Godzilla while the monster is hacking into the pentagon using ssh. that's the movie GAN, more interesting would be the zero-shot translations of epic foreign/historic films into your native culture
- temp_account_32 4y ago
  Isn't everything representable in a digital form? I think we're in the very early era of entertainment becoming commoditized to an even higher degree than it is now. I envision exactly the future as you describe: Feed a song to the AI, it spits out a completely new, whole discography from the artist complete with lyrics and album art that you can listen to infinitely. "Hey Siri, play me a series about chickens from outer space invading Earth": No problem, here's a 12 hour marathon, complete with a coherent storyline, plot twists, good acting and voice lines. The only thing that is currently limiting us is computing power, and given enough time, the barrier will be overcome. A human brain is just a series of inputs, a function that transforms them, and a series of outputs.
- peoplefromibiza 4y ago
  > So, in some channel at some time you may be able to see Brad Pitt fighting against Godzilla while the monster is hacking into the pentagon using ssh. Don't know if this is sarcasm, if it is, ignore the rest of the comment. Honestly, it sounds terrible. Good shows are well written, coherent and, most of all, narrow in scope. If an AI can write the next Better Call Saul, great. Randomly patching together tropes sounds more like kids drawings, that are interesting on an artistic point of view, maybe, given their limited knowledge of reality and narrative, but terribly boring and confusing as a form of entertainment. Unless the audience is kids, they love that stuff, for reasons we don't understand anymore as we grow up.
  - bigyikes 4y ago
    The point of the quoted remark is to illustrate the exotic possibilities of “Multiverse TV”. It’s not an example of quality content you /would/ watch, it’s merely some content you /could/ watch. Multiverse TV has everything, from Better Call Saul to zany strings of tropes.
gpm 4y ago
Huh, there's a ton of duplicates in the data set... I would have expected that it would be worthwhile to remove those. Maybe multiple descriptions of the same thing helps, but some of the duplicates have duplicated descriptions as well. Maybe deduplication happens after this step? http://laion-aesthetic.datasette.io/laion-aesthetic-6pls/images?_sort_desc=aesthetic http://laion-aesthetic.datasette.io/laion-aesthetic-6pls/ima...
- minimaxir 4y ago
  Per the project page: https://laion.ai/blog/laion-400-open-dataset/ https://laion.ai/blog/laion-400-open-dataset/ > There is a certain degree of duplication because we used URL+text as deduplication criteria. The same image with the same caption may sit at different URLs, causing duplicates. The same image with other captions is not, however, considered duplicated. I am surprised that image-to-image dupes aren't removed, though, as the cosine similarity trick the page mentions would work for that too.
  - kaibee 4y ago
    I assume having multiple captions for the same image is very helpful actually.
    - minimaxir 4y ago
      Scrolling through the sorted link from the GP, there are a few dupes with identical images and captions, so that doesn't always work either.
  - gchamonlive 4y ago
    Isn't it really expensive to dedupe images based on content? As you have to compare every image to every other image in the dataset? How could one go about deduping images? Maybe using something similar to rsync protocol? Cheap hash method, then a more expensive one, then a full comparison, maybe. Even so 2B+ images... and you are talking about saving on storage costs, mostly which is quite cheap these days.
TaylorAlexander 4y ago
"The most frequent artist in the dataset? The Painter of Light himself, Thomas Kinkade, with 9,268 images." Oh that's why it is so good at generating Thomas Kinkade style paintings! I ran a bunch of those and they looked pretty good. Some kind of garden cottage prompt with Thomas Kinkade style works very well. Good image consistency with a high success rate, few weird artifacts.
- z3c0 4y ago
  I've noticed an affinity towards "The Greats" of modern painting. I've gotten incredible results from using Dali, Picasso, Bacon, Lichtenstein, etc. My luck with slightly-less-known artists of similar styles doesn't work as well (eg Braque, Guayasamín, or Gris, as opposed to Picasso)
- Quiark 4y ago
  Can you come up with a prompt that will reproduce some of his painting almost exactly?
  - TaylorAlexander 4y ago
    Not sure.
    - dr_dshiv 4y ago
      I’m curious about this. Can it plagerize? Can GPT3? Why or why not?
NIL8 4y ago
It feels like a tsunami is coming and we have no idea how big it will be.
- deleted 4y ago
  [deleted]
- wil421 4y ago
  Like the self driving revolution? Or the bitcoin/blockchain revolution? Personally, I’m not even getting out my popcorn yet.
  - simonw 4y ago
    Very different. This stuff is genuinely useful already, and is getting more effective every day.
  - marktolson 4y ago
    Bitcoin / Blockchain doesn't have any intrinsic value other than to those who believe in it. Self driving cars (Level 4-5) are not available to the public and is still in development. This stuff is real, produces some incredible results, available to the public and advancing at a rapid rate.
    - cdata 4y ago
      The output of these models seems really impressive, but for my money the notion that it has value is undermined by the way its trainers keep "proprietary" data that is likely to be in violation of image usage rights at a large scale. What is the true value of something that can only be had at the other end of misbegotten extraction/exploitation? It seems like a similar trade-off to the one that web3 proponents are asking us to make. The apparent end-game is that we'll kill off all the true value creators - the working artists responsible for the source data - and all we'll be left with is an artifact of their works.
      - nprateem 4y ago
simonw 4y ago
If anyone is interested in the technical details, the database itself is a 4GB SQLite file which we are hosting with Datasette running on Fly. More details in our repo: https://github.com/simonw/laion-aesthetic-datasette https://github.com/simonw/laion-aesthetic-datasette Search is provided by SQLite FTS5.
- sdwr 4y ago
  The search speed is amazing!! Do you have to do a lot of pre-indexing to get it so fast?
  - simonw 4y ago
    It's SQLite's built in FTS index, nothing special on top of it. I built the index by running: sqlite-utils enable-fts data.db images text https://sqlite-utils.datasette.io/en/stable/cli.html#configuring-full-text-search https://sqlite-utils.datasette.io/en/stable/cli.html#configu...
    - sdwr 4y ago
      Are you running on anything special compute-wise? I have a budget node running Mongo, it takes almost a second to fetch a single 1MB document. Writing it out I realize it's not indexed by the attribute I'm retrieving by...
      - isoprophlex 4y ago
        Use the index, Luke!
      - snovv_crash 4y ago
        But MongoDB is webscale!
      - simonw 4y ago
        I started this on a Fly instance with 256MB of RAM and a shared CPU. This worked great when it was just a couple of people testing it. Once it started getting traffic it started running a bit slow, so I bumped it up to a 2 CPU instance with 4GB of RAM and it's been fine since then. The database file is nearly 4GB and almost all memory is being used, so I guess it all got loaded into RAM by SQLite. I'll scale it back down again in a few days time, once interest in it wanes a bit.
avocado2 4y ago
supernova87a 4y ago
My fundamental and maybe dumb question is: when is artificial intelligence / ML going to get smarter than needing a billion images to train? Sure, the achievements of ML models lately are impressive, but it's so slow in learning. We are brute forcing the DNNs it feels to me, which is not something that smacks of great achievement. You and I have never seen even 100,000 photos in our lives. Well, maybe the video stream from our eyes is a little different. But it's not a billion fundamentally different images. Is there anything I can read about why it is so slow to learn? How will it ever get faster? What next jump will fix this, or what am I missing as a lay person?
- simonw 4y ago
  I recommend looking into "transfer learning". That's where you start with an existing large model, and train a new model on top of it by feeding in new images. What's fascinating about transfer learning is that you don't need to give it a lot of new images, at all. Just a few hundred extras can create a model that's frighteningly accurate for tasks like image labeling. This is pretty much how all AI models work today. Take a look at the Stable Diffusion model card: https://github.com/CompVis/stable-diffusion/blob/main/Stable_Diffusion_v1_Model_Card.md#training https://github.com/CompVis/stable-diffusion/blob/main/Stable... They ran multiple training sessions with progressively smaller (and higher quality) images to get the final result.
- namose 4y ago
  I think the thing that’s missing is that the AI can’t train itself. If you were asked to draw a realistic x ray of a horse’s ribcage, you’d probably google image search, do some research about horse anatomy, etc, before putting pen to paper. This thing is being trained exactly once, and can’t learn dynamically. That’ll be the next step I think.
  - andreyk 4y ago
    What are you are describing is pretty much reinforcement learning (or learning with access to a query-able knowledge engine, or active learning, or all of these combined). There is work on a bunch of variations of this, but it's true that it's early days for combining it with generative systems.
wongarsu 4y ago
This is quite interesting. This really makes me wonder how much of the differences between Stable Diffusion, Dall-e 2 and MidJourney are due to different architectures and training intensity and how much is due to different datasets. For example Stable Diffusion knows much better than MidJourney what a cat looks like, MidJourney knows what a Hacker Cat looks like, while Stable Diffusion doesn't (you can tell it to make a cat in a hoodie in front of a laptop, but it won't come up with that on its own). Meanwhile for landscapes Stable Diffusion seems to have no problem with imagination. How much of that is simply due to blindspots in the training data?
guelo 4y ago
The next internet crawl is going to have thousands of (properly labeled) AI generated images. I wonder if that could throw these algorithms into a bad feedback spiral. Though I guess there are already classifiers that can be used to exclude AI generated images. The goal is to profit off of free human labor after all.
- practice9 4y ago
  > The goal is to profit off of free human labor after all. this is too inflammatory in my opinion. - some works are in public domain - tech companies have profited from creators for a long time. I'm sure some arrangement could be made for profit sharing for artists who care about money, but it's too early for that (no profits, I'm sure most companies are losing money on AI art) - some artists care about art or fame more than money. Their art will not be devalued by AI, if anything, constant usage of their names in prompts is going to make them massively popular and direct people to source material or merch, which they may buy. - some artists are dead and don't care anymore. Their "estate" vulnerable to takeovers by "long lost but recently found" relatives, who don't care about art itself, only about money. Many such stories. One example, albeit not in paintings but in music is Jimi Hendrix Estate. They used to do copyright strikes on YouTube in order to remove fan-made compilations of rare material (cleaned up sound of live concerts, multiple sources mixed into one etc.), without intentions to ever release an alternative.
karpathy 4y ago
Data is an excellent place to look at to get a sense of where the model is likely to work or not (what kinds of images), and for prompt design ideas because, roughly speaking, the probability of something working well is proportional to its frequency (or of things very similar to it) in the data. The story is more complex though because the data can often be quite far away from actual neural net training due to preprocessing steps, data augmentations, oversampling settings (it's not uncommon to not sample data uniformly during training), etc. So my favorite place to scrutinize is to build a "batch explorer": During training of the network one dumps batches into pickles immediately before the forward pass of the neural net, then writes a separate explorer that loads the pickles and visualizes them to "see exactly what the neural net sees" during training. Ideally one then spends some quality time (~hours) looking through batches to get a qualitiative sense of what is likely to work or not work and how well. Of course this is also very useful for debugging, as many bugs can be present in the data preprocessing pipeline. But a batch explorer is harder to obtain here because you'd need the full training data/code/settings.
- jimsimmons 4y ago
  With big generative models, seeing data even once is more than sufficient to memorize it. So your claim that performance relates to frequency is not exactly correct. The whole point of this model class is that one can learn one word from one sample, another pixel from another one and so on to master the domain. The emergent, non-trivial generalization is what makes them so fascinating. There is no simple, linear/first order relationship with data and behaviour. Case in point: GPT3 can do few-shot learning despite not having used any explicit few-shot formatted data during training. Not saying you are wrong, but the story is not as simple as simple supervised learning with small datasets
  - galangalalgol 4y ago
    What does that say about how these models will behave as an increasingly large portion of their training data is outputs from similar models? Our curation of the outputs will hopefully help. And if one image really is enough, perhaps the smaller number of human created images will be sufficient to inject new stuff rather than stagnating?
egypturnash 4y ago
Aaaand the estimated percentage of images released under a CC license, or public domain, iiiiis...?
- deleted 4y ago
  [deleted]
ISL 4y ago
That inclusion of Mickey in the model is waving a red flag in front of an impressive bull.
- tough 4y ago
  I was just thinking yesterday if Mockey The Rat would fly as homage/deriv mickey has some good 70 years already no? Copyright will die at the hands of ML am afraid
deleted 4y ago
[deleted]
thrdbndndn 4y ago
Why not just add a shortcut to search for the related images in these artist count table etc.? Also, a UI issue: the sorting arrow feels wrong: https://i.imgur.com/uyUXAXy.png https://i.imgur.com/uyUXAXy.png The norm is that when the arrow is pointing down, the data are currently sorting descendingly (I'm aware you can interpret it as "what will happen if you click it", but the norm is to show what currently is using.)
- simonw 4y ago
  That's the exact problem: should the button show what will happen, or what is currently happening? When I designed that feature I looked at a bunch of systems and found examples of arrows in both directions.
  - thrdbndndn 4y ago
    At least Excel and Google Sheets both display it the way I described. Adding Apple Numbers, which I never use unfortunately, I think that should cover like 90% of actual use cases to be considered as "convention".
thrdbndndn 4y ago
> but often impossible with DALL-E 2, as you can see in this Mickey Mouse example from my previous post > “realistic 3d rendering of mickey mouse working on a vintage computer doing his taxes” on DALL·E 2 (left) vs. Stable Diffusion (right) Well, but the mickey mouse in the right isn't "realistic", or even 3D. It's straight up just a 2D Mickey image pasted there.
ks2048 4y ago
They need a feature that for a given generation, shows the nearest image in the training set. It is clearly doing more than “memorizing”, but for some “normal” queries, how do you know the output isn’t very similar to some training image? That could have legal implications.
- speedgoose 4y ago
  You could try the reverse search in a images search engine. Google or Bing support that for example.
- mkl 4y ago
  I don't think anyone has a definition of "nearest" that could accomplish that in general. Comparing pixel data is easy, but comparing the subjects portrayed and how is much harder to pin down.
ryandv 4y ago
- deleted 4y ago
  [deleted]
Imnimo 4y ago
It is interesting that there are at least a few images in the dataset that were generated by previous diffusion methods: https://laion-aesthetic.datasette.io/laion-aesthetic-6pls/images?_search=CLIP+guided+diffusion&_sort=rowid https://laion-aesthetic.datasette.io/laion-aesthetic-6pls/im... (Surely there are many more that don't have this specific label).
- htrp 4y ago
  Turtles all the way down
cakeface 4y ago
In which I run `order by punsafe desc` and immediately regret it.
- nprateem 4y ago
  Although it did show me that my masseuse isn't as skilled as I thought
nbzso 4y ago
Thinking over this "magical" tech. A distinctive painting style is one of the biggest goals for an artist. Producing unique stroke and clear expression takes years of hard work and experimentation. An artist decides to sell prints of an expensive artwork, he or she publishes a photo on their website. AI scrapers get the images in data set update. Game over for the artist. I hope for a class action over this training data sets. I get that kids have fun with the new photoshop filters. I get that software is eating "the world" but someone must wake up an push the kill switch. It is possible.
- sennight 4y ago
  Sounds like a hopeless protectionist endeavor, reminiscent of cartoonish Keynesian economics busywork: "We can't permit development of adding machines, what about all the hard work people have put into memorizing multiplication tables?!".
  - nbzso 4y ago
    No. Sounds like common sense. Not popular in the tech community nowadays. Data is the new "petrol". Since when the petrol is free of charge? People in mass are clueless. If you use my "human" accomplishments as an energy source, you must pay me. Period.
    - sennight 4y ago
      > Since when the petrol is free of charge? At around the same time that it actually resembles "the new data", or maybe even shares a single quality with it? Being a physical object, it is bound by physical properties... like scarcity. Data, being an abstract concept, suffers no such constraint. Same story for whatever artistic technique you've imagined to be not only valuable, but novel to all of human experience and wholly owned by you and you alone. Your valuation of your worth and that of your labor is laughably overinflated and the market is telling you so. Period.
      - nbzso 4y ago
revskill 4y ago
It's hard for user to produce expected effects of result without clear guidance on correct grammar, contextual information,...
r3trohack3r 4y ago
This has been an incredibly helpful tool today exploring Stable Diffusion. I'm starting to realize Stable Diffusion doesn't understand many words, but it's hard to tell which words are causing it problems when engineering a prompt. Searching this dataset for a term is a great way to tell whether Stable Diffusion is likely to "understand" what I mean when I say that term; if there are few results, or if the results aren't really representative of what I mean, Stable Diffusion is likely to produce garbage outputs for those terms.
dmitriid 4y ago
> Nearly half of the images, about 47%, were sourced from only 100 domains, with the largest number of images coming from Pinterest This makes me vaguely uneasy. All these models and tools are almost exclusively "western".
- tough 4y ago
  > https://github.com/gradio-app/gradio https://github.com/gradio-app/gradio Plenty of asian / artists / styles on the datasets no?
fareesh 4y ago
How would one go about adding more data into the dataset? Would one need to retrain the entire dataset? Or is there typically a way to just add an incremental batch?
epups 4y ago
As expected, very few NSFW images were included in the training set, according to this. They are more afraid of showing a penis than showing Mickey Mouse.
rom1504 4y ago
Hi, laion5b author here, Nice tool! You can also explore the dataset there https://rom1504.github.io/clip-retrieval/ https://rom1504.github.io/clip-retrieval/ Thanks to approximate knn, it's possible to query and explore that 5B datasets with only 2TB of local storage, anyone can download the knn index and metadata to run that locally too. Regarding duplicates, indeed it's an interesting topic! Laion5b deduplicated samples by url+text, but not by image. To deduplicate by image you need to have an efficient way to compute whether image a and b are the same. An idea to do that is to compute an hash based on clip embeddings. A further idea would be to train a network actually good at dedup and not only similarity by training on positive and negative pairs, eg with triple loss. Here's my plan on the topic https://docs.google.com/document/d/1AryWpV0dD_r9x82I_quUzBuRyzDotL_HHnKuNB9H3Zc/edit?usp=drivesdk https://docs.google.com/document/d/1AryWpV0dD_r9x82I_quUzBuR... If anyone is interested to participate, I'd be happy to guide them to do that. This is an open effort, just join laion discord server and let's talk.
- yreg 4y ago
  I have been using the rom1504 clip retrieval tool[0] up until now, but the Datasette browser[1] seems much better for Stable Diffusion users. When my prompt isn't working, I often want to check whether the concepts I use are even present in the dataset. For example, inputting `Jony Ive` returns pictures of Jony Ive in Datasette and pictures of apples and dolls in clip retrieval. (I know laion 5B is not the same as laion aesthetic 6+, but that's a lesser issue.) [0] - https://rom1504.github.io/clip-retrieval/ https://rom1504.github.io/clip-retrieval/ [1] - https://laion-aesthetic.datasette.io/laion-aesthetic-6pls/images https://laion-aesthetic.datasette.io/laion-aesthetic-6pls/im...
  - rom1504 4y ago
    This is due to the aesthetic scoring in the UI. Simply disable it if you want precise results rather than aesthetic ones. It works for your example I guess I'll disable it by default since it seems to confuse people
    - rom1504 4y ago
withinboredom 4y ago
NSFW is entertaining. It tends to think “knobs” in the prompt mean “female breasts” which is annoying.
deleted 4y ago
[deleted]
brohee 4y ago
> Strangely, enormously popular internet personalities like David Dobrik, Addison Rae, Charli D’Amelio, Dixie D’Amelio, and MrBeast don’t appear in the captions from the dataset at all Self awareness here would have lead to the removal of "enormously popular".
lovelearning 4y ago
> Unsurprisingly, a large number came from stock image sites...Adobe Stock’s...iStockPhoto...Shutterstock Are they ok with their stock photos being used to train a service that's likely to bite into their stock photo business?
- PeterisP 4y ago
  The general understanding is that noone cares if they're ok with that as training a model (even if a competing service) is not among the exclusive rights of copyright owners where their permission is required - if you have a legally obtained copy, you can use it as you want (except for the specific explicitly enumerated use cases that copyright law awards exclusively to the copyright holder) no matter if the copyright holder likes it or not.

446 points by detaro 4y ago

So excellent. Flipping the story we see all the time on it's head. AI's quasi-mystical powers are endless spectacle. Taking a look through the other side of the looking glass is vastly overdue. Amazing work. Just just just starting to scratch the surface. 2% of the data gathered, sources identified. These are a couple of sites we can now source as the primary powerers of AI. Barely reviewed, dived into in terms of the content itself. We have so little sense & appreciation for what lurks beneath, but this is a go.

benreesman 4y ago
It is an unambiguous social necessity to demystify these things. In a world where the lay public didn’t really know about photoshop, photoshop would be a terrifying weapon. Likewise modern ML is for the most part mysterious and/or menacing because it’s opaque and arcane and mostly controlled by big corporate R&D labs. Get some charismatic science popularizers out there teaching people how it works, all the sudden not such a big scary thing.

lmarcos 4y ago

I always had the crazy idea of "infinite entertainment": somehow we manage to "tap" into the multiverse and are able to watch TV from countless of planets/universes (I think Rick and Morty did something similar). So, in some channel at some time you may be able to see Brad Pitt fighting against Godzilla while the monster is hacking into the pentagon using ssh. Highly improbable, but in the multiverse TV everything is possible. Now I think we don't need the multiverse for that. Give this AI technology a few years and you'll have streaming services a la Netflix where you provide the prompt to create your own movie. What the hell, people will vote "best movie" among the millions submitted by other people. We'll be movie producers like we are nowadays YouTubers. Overabundance of high quality material and so little time to watch them all. Same goes for books, music and everything else that is digital (even software?).

gfody 4y ago
> So, in some channel at some time you may be able to see Brad Pitt fighting against Godzilla while the monster is hacking into the pentagon using ssh. that's the movie GAN, more interesting would be the zero-shot translations of epic foreign/historic films into your native culture
temp_account_32 4y ago
Isn't everything representable in a digital form? I think we're in the very early era of entertainment becoming commoditized to an even higher degree than it is now. I envision exactly the future as you describe: Feed a song to the AI, it spits out a completely new, whole discography from the artist complete with lyrics and album art that you can listen to infinitely. "Hey Siri, play me a series about chickens from outer space invading Earth": No problem, here's a 12 hour marathon, complete with a coherent storyline, plot twists, good acting and voice lines. The only thing that is currently limiting us is computing power, and given enough time, the barrier will be overcome. A human brain is just a series of inputs, a function that transforms them, and a series of outputs.
peoplefromibiza 4y ago
> So, in some channel at some time you may be able to see Brad Pitt fighting against Godzilla while the monster is hacking into the pentagon using ssh. Don't know if this is sarcasm, if it is, ignore the rest of the comment. Honestly, it sounds terrible. Good shows are well written, coherent and, most of all, narrow in scope. If an AI can write the next Better Call Saul, great. Randomly patching together tropes sounds more like kids drawings, that are interesting on an artistic point of view, maybe, given their limited knowledge of reality and narrative, but terribly boring and confusing as a form of entertainment. Unless the audience is kids, they love that stuff, for reasons we don't understand anymore as we grow up.
- bigyikes 4y ago
  The point of the quoted remark is to illustrate the exotic possibilities of “Multiverse TV”. It’s not an example of quality content you /would/ watch, it’s merely some content you /could/ watch. Multiverse TV has everything, from Better Call Saul to zany strings of tropes.

gpm 4y ago

Huh, there's a ton of duplicates in the data set... I would have expected that it would be worthwhile to remove those. Maybe multiple descriptions of the same thing helps, but some of the duplicates have duplicated descriptions as well. Maybe deduplication happens after this step? http://laion-aesthetic.datasette.io/laion-aesthetic-6pls/images?_sort_desc=aesthetic http://laion-aesthetic.datasette.io/laion-aesthetic-6pls/ima...

minimaxir 4y ago
Per the project page: https://laion.ai/blog/laion-400-open-dataset/ https://laion.ai/blog/laion-400-open-dataset/ > There is a certain degree of duplication because we used URL+text as deduplication criteria. The same image with the same caption may sit at different URLs, causing duplicates. The same image with other captions is not, however, considered duplicated. I am surprised that image-to-image dupes aren't removed, though, as the cosine similarity trick the page mentions would work for that too.
- kaibee 4y ago
  I assume having multiple captions for the same image is very helpful actually.
  - minimaxir 4y ago
    Scrolling through the sorted link from the GP, there are a few dupes with identical images and captions, so that doesn't always work either.
- gchamonlive 4y ago
  Isn't it really expensive to dedupe images based on content? As you have to compare every image to every other image in the dataset? How could one go about deduping images? Maybe using something similar to rsync protocol? Cheap hash method, then a more expensive one, then a full comparison, maybe. Even so 2B+ images... and you are talking about saving on storage costs, mostly which is quite cheap these days.

TaylorAlexander 4y ago

"The most frequent artist in the dataset? The Painter of Light himself, Thomas Kinkade, with 9,268 images." Oh that's why it is so good at generating Thomas Kinkade style paintings! I ran a bunch of those and they looked pretty good. Some kind of garden cottage prompt with Thomas Kinkade style works very well. Good image consistency with a high success rate, few weird artifacts.

z3c0 4y ago
I've noticed an affinity towards "The Greats" of modern painting. I've gotten incredible results from using Dali, Picasso, Bacon, Lichtenstein, etc. My luck with slightly-less-known artists of similar styles doesn't work as well (eg Braque, Guayasamín, or Gris, as opposed to Picasso)
Quiark 4y ago
Can you come up with a prompt that will reproduce some of his painting almost exactly?
- TaylorAlexander 4y ago
  Not sure.
  - dr_dshiv 4y ago
    I’m curious about this. Can it plagerize? Can GPT3? Why or why not?

NIL8 4y ago

It feels like a tsunami is coming and we have no idea how big it will be.

deleted 4y ago
[deleted]
wil421 4y ago
Like the self driving revolution? Or the bitcoin/blockchain revolution? Personally, I’m not even getting out my popcorn yet.
- simonw 4y ago
  Very different. This stuff is genuinely useful already, and is getting more effective every day.
- marktolson 4y ago
  Bitcoin / Blockchain doesn't have any intrinsic value other than to those who believe in it. Self driving cars (Level 4-5) are not available to the public and is still in development. This stuff is real, produces some incredible results, available to the public and advancing at a rapid rate.
  - cdata 4y ago
    The output of these models seems really impressive, but for my money the notion that it has value is undermined by the way its trainers keep "proprietary" data that is likely to be in violation of image usage rights at a large scale. What is the true value of something that can only be had at the other end of misbegotten extraction/exploitation? It seems like a similar trade-off to the one that web3 proponents are asking us to make. The apparent end-game is that we'll kill off all the true value creators - the working artists responsible for the source data - and all we'll be left with is an artifact of their works.
    - nprateem 4y ago

simonw 4y ago

If anyone is interested in the technical details, the database itself is a 4GB SQLite file which we are hosting with Datasette running on Fly. More details in our repo: https://github.com/simonw/laion-aesthetic-datasette https://github.com/simonw/laion-aesthetic-datasette Search is provided by SQLite FTS5.

sdwr 4y ago
The search speed is amazing!! Do you have to do a lot of pre-indexing to get it so fast?
- simonw 4y ago
  It's SQLite's built in FTS index, nothing special on top of it. I built the index by running: sqlite-utils enable-fts data.db images text https://sqlite-utils.datasette.io/en/stable/cli.html#configuring-full-text-search https://sqlite-utils.datasette.io/en/stable/cli.html#configu...
  - sdwr 4y ago
    Are you running on anything special compute-wise? I have a budget node running Mongo, it takes almost a second to fetch a single 1MB document. Writing it out I realize it's not indexed by the attribute I'm retrieving by...
    - isoprophlex 4y ago
      Use the index, Luke!
    - snovv_crash 4y ago
      But MongoDB is webscale!
    - simonw 4y ago
      I started this on a Fly instance with 256MB of RAM and a shared CPU. This worked great when it was just a couple of people testing it. Once it started getting traffic it started running a bit slow, so I bumped it up to a 2 CPU instance with 4GB of RAM and it's been fine since then. The database file is nearly 4GB and almost all memory is being used, so I guess it all got loaded into RAM by SQLite. I'll scale it back down again in a few days time, once interest in it wanes a bit.

avocado2 4y ago

supernova87a 4y ago

My fundamental and maybe dumb question is: when is artificial intelligence / ML going to get smarter than needing a billion images to train? Sure, the achievements of ML models lately are impressive, but it's so slow in learning. We are brute forcing the DNNs it feels to me, which is not something that smacks of great achievement. You and I have never seen even 100,000 photos in our lives. Well, maybe the video stream from our eyes is a little different. But it's not a billion fundamentally different images. Is there anything I can read about why it is so slow to learn? How will it ever get faster? What next jump will fix this, or what am I missing as a lay person?

simonw 4y ago
I recommend looking into "transfer learning". That's where you start with an existing large model, and train a new model on top of it by feeding in new images. What's fascinating about transfer learning is that you don't need to give it a lot of new images, at all. Just a few hundred extras can create a model that's frighteningly accurate for tasks like image labeling. This is pretty much how all AI models work today. Take a look at the Stable Diffusion model card: https://github.com/CompVis/stable-diffusion/blob/main/Stable_Diffusion_v1_Model_Card.md#training https://github.com/CompVis/stable-diffusion/blob/main/Stable... They ran multiple training sessions with progressively smaller (and higher quality) images to get the final result.
namose 4y ago
I think the thing that’s missing is that the AI can’t train itself. If you were asked to draw a realistic x ray of a horse’s ribcage, you’d probably google image search, do some research about horse anatomy, etc, before putting pen to paper. This thing is being trained exactly once, and can’t learn dynamically. That’ll be the next step I think.
- andreyk 4y ago
  What are you are describing is pretty much reinforcement learning (or learning with access to a query-able knowledge engine, or active learning, or all of these combined). There is work on a bunch of variations of this, but it's true that it's early days for combining it with generative systems.

wongarsu 4y ago

This is quite interesting. This really makes me wonder how much of the differences between Stable Diffusion, Dall-e 2 and MidJourney are due to different architectures and training intensity and how much is due to different datasets. For example Stable Diffusion knows much better than MidJourney what a cat looks like, MidJourney knows what a Hacker Cat looks like, while Stable Diffusion doesn't (you can tell it to make a cat in a hoodie in front of a laptop, but it won't come up with that on its own). Meanwhile for landscapes Stable Diffusion seems to have no problem with imagination. How much of that is simply due to blindspots in the training data?

guelo 4y ago

The next internet crawl is going to have thousands of (properly labeled) AI generated images. I wonder if that could throw these algorithms into a bad feedback spiral. Though I guess there are already classifiers that can be used to exclude AI generated images. The goal is to profit off of free human labor after all.

practice9 4y ago
> The goal is to profit off of free human labor after all. this is too inflammatory in my opinion. - some works are in public domain - tech companies have profited from creators for a long time. I'm sure some arrangement could be made for profit sharing for artists who care about money, but it's too early for that (no profits, I'm sure most companies are losing money on AI art) - some artists care about art or fame more than money. Their art will not be devalued by AI, if anything, constant usage of their names in prompts is going to make them massively popular and direct people to source material or merch, which they may buy. - some artists are dead and don't care anymore. Their "estate" vulnerable to takeovers by "long lost but recently found" relatives, who don't care about art itself, only about money. Many such stories. One example, albeit not in paintings but in music is Jimi Hendrix Estate. They used to do copyright strikes on YouTube in order to remove fan-made compilations of rare material (cleaned up sound of live concerts, multiple sources mixed into one etc.), without intentions to ever release an alternative.

karpathy 4y ago

Data is an excellent place to look at to get a sense of where the model is likely to work or not (what kinds of images), and for prompt design ideas because, roughly speaking, the probability of something working well is proportional to its frequency (or of things very similar to it) in the data. The story is more complex though because the data can often be quite far away from actual neural net training due to preprocessing steps, data augmentations, oversampling settings (it's not uncommon to not sample data uniformly during training), etc. So my favorite place to scrutinize is to build a "batch explorer": During training of the network one dumps batches into pickles immediately before the forward pass of the neural net, then writes a separate explorer that loads the pickles and visualizes them to "see exactly what the neural net sees" during training. Ideally one then spends some quality time (~hours) looking through batches to get a qualitiative sense of what is likely to work or not work and how well. Of course this is also very useful for debugging, as many bugs can be present in the data preprocessing pipeline. But a batch explorer is harder to obtain here because you'd need the full training data/code/settings.

jimsimmons 4y ago
With big generative models, seeing data even once is more than sufficient to memorize it. So your claim that performance relates to frequency is not exactly correct. The whole point of this model class is that one can learn one word from one sample, another pixel from another one and so on to master the domain. The emergent, non-trivial generalization is what makes them so fascinating. There is no simple, linear/first order relationship with data and behaviour. Case in point: GPT3 can do few-shot learning despite not having used any explicit few-shot formatted data during training. Not saying you are wrong, but the story is not as simple as simple supervised learning with small datasets
- galangalalgol 4y ago
  What does that say about how these models will behave as an increasingly large portion of their training data is outputs from similar models? Our curation of the outputs will hopefully help. And if one image really is enough, perhaps the smaller number of human created images will be sufficient to inject new stuff rather than stagnating?

egypturnash 4y ago

Aaaand the estimated percentage of images released under a CC license, or public domain, iiiiis...?

deleted 4y ago
[deleted]

ISL 4y ago

That inclusion of Mickey in the model is waving a red flag in front of an impressive bull.

tough 4y ago
I was just thinking yesterday if Mockey The Rat would fly as homage/deriv mickey has some good 70 years already no? Copyright will die at the hands of ML am afraid

deleted 4y ago

[deleted]

thrdbndndn 4y ago

Why not just add a shortcut to search for the related images in these artist count table etc.? Also, a UI issue: the sorting arrow feels wrong: https://i.imgur.com/uyUXAXy.png https://i.imgur.com/uyUXAXy.png The norm is that when the arrow is pointing down, the data are currently sorting descendingly (I'm aware you can interpret it as "what will happen if you click it", but the norm is to show what currently is using.)

simonw 4y ago
That's the exact problem: should the button show what will happen, or what is currently happening? When I designed that feature I looked at a bunch of systems and found examples of arrows in both directions.
- thrdbndndn 4y ago
  At least Excel and Google Sheets both display it the way I described. Adding Apple Numbers, which I never use unfortunately, I think that should cover like 90% of actual use cases to be considered as "convention".

thrdbndndn 4y ago

> but often impossible with DALL-E 2, as you can see in this Mickey Mouse example from my previous post > “realistic 3d rendering of mickey mouse working on a vintage computer doing his taxes” on DALL·E 2 (left) vs. Stable Diffusion (right) Well, but the mickey mouse in the right isn't "realistic", or even 3D. It's straight up just a 2D Mickey image pasted there.

ks2048 4y ago

They need a feature that for a given generation, shows the nearest image in the training set. It is clearly doing more than “memorizing”, but for some “normal” queries, how do you know the output isn’t very similar to some training image? That could have legal implications.

speedgoose 4y ago
You could try the reverse search in a images search engine. Google or Bing support that for example.
mkl 4y ago
I don't think anyone has a definition of "nearest" that could accomplish that in general. Comparing pixel data is easy, but comparing the subjects portrayed and how is much harder to pin down.

ryandv 4y ago

deleted 4y ago
[deleted]

Imnimo 4y ago

It is interesting that there are at least a few images in the dataset that were generated by previous diffusion methods: https://laion-aesthetic.datasette.io/laion-aesthetic-6pls/images?_search=CLIP+guided+diffusion&_sort=rowid https://laion-aesthetic.datasette.io/laion-aesthetic-6pls/im... (Surely there are many more that don't have this specific label).

htrp 4y ago
Turtles all the way down

cakeface 4y ago

In which I run `order by punsafe desc` and immediately regret it.

nprateem 4y ago
Although it did show me that my masseuse isn't as skilled as I thought

nbzso 4y ago

Thinking over this "magical" tech. A distinctive painting style is one of the biggest goals for an artist. Producing unique stroke and clear expression takes years of hard work and experimentation. An artist decides to sell prints of an expensive artwork, he or she publishes a photo on their website. AI scrapers get the images in data set update. Game over for the artist. I hope for a class action over this training data sets. I get that kids have fun with the new photoshop filters. I get that software is eating "the world" but someone must wake up an push the kill switch. It is possible.

sennight 4y ago
Sounds like a hopeless protectionist endeavor, reminiscent of cartoonish Keynesian economics busywork: "We can't permit development of adding machines, what about all the hard work people have put into memorizing multiplication tables?!".
- nbzso 4y ago
  No. Sounds like common sense. Not popular in the tech community nowadays. Data is the new "petrol". Since when the petrol is free of charge? People in mass are clueless. If you use my "human" accomplishments as an energy source, you must pay me. Period.
  - sennight 4y ago
    > Since when the petrol is free of charge? At around the same time that it actually resembles "the new data", or maybe even shares a single quality with it? Being a physical object, it is bound by physical properties... like scarcity. Data, being an abstract concept, suffers no such constraint. Same story for whatever artistic technique you've imagined to be not only valuable, but novel to all of human experience and wholly owned by you and you alone. Your valuation of your worth and that of your labor is laughably overinflated and the market is telling you so. Period.
    - nbzso 4y ago

revskill 4y ago

It's hard for user to produce expected effects of result without clear guidance on correct grammar, contextual information,...

r3trohack3r 4y ago

This has been an incredibly helpful tool today exploring Stable Diffusion. I'm starting to realize Stable Diffusion doesn't understand many words, but it's hard to tell which words are causing it problems when engineering a prompt. Searching this dataset for a term is a great way to tell whether Stable Diffusion is likely to "understand" what I mean when I say that term; if there are few results, or if the results aren't really representative of what I mean, Stable Diffusion is likely to produce garbage outputs for those terms.

dmitriid 4y ago

> Nearly half of the images, about 47%, were sourced from only 100 domains, with the largest number of images coming from Pinterest This makes me vaguely uneasy. All these models and tools are almost exclusively "western".

tough 4y ago
> https://github.com/gradio-app/gradio https://github.com/gradio-app/gradio Plenty of asian / artists / styles on the datasets no?

fareesh 4y ago

How would one go about adding more data into the dataset? Would one need to retrain the entire dataset? Or is there typically a way to just add an incremental batch?

epups 4y ago

As expected, very few NSFW images were included in the training set, according to this. They are more afraid of showing a penis than showing Mickey Mouse.

rom1504 4y ago

Hi, laion5b author here, Nice tool! You can also explore the dataset there https://rom1504.github.io/clip-retrieval/ https://rom1504.github.io/clip-retrieval/ Thanks to approximate knn, it's possible to query and explore that 5B datasets with only 2TB of local storage, anyone can download the knn index and metadata to run that locally too. Regarding duplicates, indeed it's an interesting topic! Laion5b deduplicated samples by url+text, but not by image. To deduplicate by image you need to have an efficient way to compute whether image a and b are the same. An idea to do that is to compute an hash based on clip embeddings. A further idea would be to train a network actually good at dedup and not only similarity by training on positive and negative pairs, eg with triple loss. Here's my plan on the topic https://docs.google.com/document/d/1AryWpV0dD_r9x82I_quUzBuRyzDotL_HHnKuNB9H3Zc/edit?usp=drivesdk https://docs.google.com/document/d/1AryWpV0dD_r9x82I_quUzBuR... If anyone is interested to participate, I'd be happy to guide them to do that. This is an open effort, just join laion discord server and let's talk.

yreg 4y ago
I have been using the rom1504 clip retrieval tool[0] up until now, but the Datasette browser[1] seems much better for Stable Diffusion users. When my prompt isn't working, I often want to check whether the concepts I use are even present in the dataset. For example, inputting `Jony Ive` returns pictures of Jony Ive in Datasette and pictures of apples and dolls in clip retrieval. (I know laion 5B is not the same as laion aesthetic 6+, but that's a lesser issue.) [0] - https://rom1504.github.io/clip-retrieval/ https://rom1504.github.io/clip-retrieval/ [1] - https://laion-aesthetic.datasette.io/laion-aesthetic-6pls/images https://laion-aesthetic.datasette.io/laion-aesthetic-6pls/im...
- rom1504 4y ago
  This is due to the aesthetic scoring in the UI. Simply disable it if you want precise results rather than aesthetic ones. It works for your example I guess I'll disable it by default since it seems to confuse people
  - rom1504 4y ago

withinboredom 4y ago

NSFW is entertaining. It tends to think “knobs” in the prompt mean “female breasts” which is annoying.

deleted 4y ago

[deleted]

brohee 4y ago

> Strangely, enormously popular internet personalities like David Dobrik, Addison Rae, Charli D’Amelio, Dixie D’Amelio, and MrBeast don’t appear in the captions from the dataset at all Self awareness here would have lead to the removal of "enormously popular".

lovelearning 4y ago

> Unsurprisingly, a large number came from stock image sites...Adobe Stock’s...iStockPhoto...Shutterstock Are they ok with their stock photos being used to train a service that's likely to bite into their stock photo business?

PeterisP 4y ago
The general understanding is that noone cares if they're ok with that as training a model (even if a competing service) is not among the exclusive rights of copyright owners where their permission is required - if you have a legally obtained copy, you can use it as you want (except for the specific explicitly enumerated use cases that copyright law awards exclusively to the copyright holder) no matter if the copyright holder likes it or not.