17 ms·
Laion-5B: A new era of open large-scale multi-modal datasets
- gillesjacobs 4y agoGood to see open data and open models become a thing, I hope this trend will continue and open AI will triumph like open source software did.
- jerpint 4y agoLAION is arguably as important as imagenet was in the early 2010s
- SubiculumCode 4y agoWhat makes this multimodal, labels?
- ShamelessC 4y agoOne mode is natural language, the other is imagery. It is combination becuse the model will learn statistical associations between the modes e.g. "text to image", "voice to text". Within these respective modes are even more subgroups e.g. language translation, audio diarization. For sd you can consider animation and photographs as separate modes the model has to learn. Although the language is fuzzy and im not being statistically rigorous as it is a weak point of mine.
- minimaxir 4y agoFor practical context, Stable Diffusion 2.X was trained on LAION-5B as opposed to LAION-400M for Stable Diffusion 1.X. At the least, Stable Diffusion 2.X is better at pain points of image generation such as text legibility and hands, potentially due to having more data points.
- Terretta 4y agoProblem with hands is probability. It's more probable a finger has a finger on both sides of it than not. So the model diffuses lots of adjacent fingers.
- stavros 4y agoBut that's the same for everything that has structure. A small section of an arm is much more likely to have another small section of an arm next to it than to have a hand, yet SD's arms are usually well-proportioned.
- sdenton4 4y agoThere's a lot of loooooong necks, though.
- alar44 4y agoNo the problem with fingers is that they resemble hotdogs and the AI really likes hotdogs so you get a lot of fingers. I can make things up too!
- satvikpendem 4y agoSD 2 also removed quite a lot of images of humans due to their fear of people generating CSAM, so the quality actually has gotten worse for anything resembling humans than SD 1.
- astrange 4y ago2.0 removed too many of them due to a bug in the NSFW filter. 2.1+ should be better again. But they’re harder to control without negative prompting.
- in3d 4y agoThis is incorrect. Stable Diffusion 1.x was trained on "laion-improved-aesthetics" (a subset of laion2B-en).
- minimaxir 4y agoDouble checked and both the initial comment and the correction are incorrect: the original v1.1 was trained on LAION-2B, then subsequent versions were finetuned on the aestethics subset. Either way, the main point is the same: more training data gives better results. https://github.com/CompVis/stable-diffusion#weights https://github.com/CompVis/stable-diffusion#weights
- abeppu 4y agoSo the core of the dataset is image _URLs_ and text captions. 1. From a reproducibility perspective, isn't this kinda brittle in that even without malicious intent, some of those images will no longer be available when other researchers attempt to download them? 2. From a resilience perspective, if your site has some of the images in the dataset, could you swap in another image with the correct dimensions. Could you poison or skew the model in any interesting ways?
- alsodumb 4y agoThe key is the scale of the dataset. Both the points you mention become irrelevant for a large dataset because 1) The chance that a significant percentage of the images become unavailable is low. Also, training on such a big dataset means your model generalizes well and is usually robust. 2) Again, you would need to inject adversarial/malicious images to a significant number of those links in the dataset for it to have actual impact on trained model. Again, unlikely.
- abeppu 4y agoFor point 1 ... it depends on the timescale. In the fullness of time, surely a significant portion of images will be unavailable. From the perspective of allowing other researchers to work from the "same" baseline "today", this is likely good enough. In a generation from now, if someone wants to reproduce results from some landmark model trained against this dataset, we'd have problems. In other fields where people publish or share their datasets, would this be considered sufficient? For point 2, I think it's possible that for some narrow topics, some domains have a significant share of images. I think these can affect the model, which is in part why they give special attention to watermarking. Suppose instead of merely watermarking images, for every image on my large collegiate track and field website I make sure someone is wearing a garment with a visible Nike swoosh. Can I skew the model towards associating Nike with the sport? I think this kind of thing may be achievable for niche areas.
- deleted 4y ago[deleted]
- 4y ago
- lajamerr 4y agoWhile impressive number of images today. I believe this will be an underwhelming amount of images compared to what models are trained on in the future. This is an incomplete analogy but from the time a baby is born that baby will have seen 1,892,160,000 frames of data per eye 3,784,320,000 frames in a year. That baby practically knows nothing about the world still.
- minimaxir 4y agoMost of those frames are redundant.
- bena 4y agoAnd unclassified. And of poor quality. Babies have a much harder task. They have to construct a corpus of knowledge from absolutely nothing.
- the8472 4y agoThe upside is that babies get to interact with the environment they're training on. Image models can't move the camera a few cm to the right if they're interested in the perspective of a particular scene.
- trasz2 4y agoHow do we know they start from nothing?
- CamperBob2 4y agoIn fact, we're pretty sure that they don't "start from nothing." E.g., https://en.wikipedia.org/wiki/The_Language_Instinct https://en.wikipedia.org/wiki/The_Language_Instinct
- bena 4y agoWe're not pretty sure of anything e.g. https://en.wikipedia.org/wiki/Educating_Eve https://en.wikipedia.org/wiki/Educating_Eve
- deleted 4y ago[deleted]
- satvikpendem 4y agoIf you've ever actually looked into the Laion datasets, you'll notice that they are hot garbage, in that the captions often don't even correlate to what the image is about, and the images are often low quality, bad cropped and so on. There are other datasets being developed that use high quality images that are manually labeled by humans, such as by Unstable Diffusion who's having a Kickstarter right now [0]. They say they will be able to get a much more high quality model due to such high quality images and captioning, so we'll see. They also want to make the model and code entirely open source rather than the license that Stable Diffusion has which is not open source (it has many restrictions, enforceable or not, on the images made). [0] https://www.kickstarter.com/projects/unstablediffusion/unstable-diffusion-unrestricted-ai-art-powered-by-the-crowd https://www.kickstarter.com/projects/unstablediffusion/unsta...
- infinityio 4y agoObviously there would be limits as to how much could be manually reviewed by hand (if 1000 people reviewed 1000 images each, only 0.02% of the images would be reviewed assuming no overlap was required), but I wonder if there would be any benefit to attempting to crowdsource captions for the dataset for the worst available images
- GaggiX 4y agoIt is not possible to manually label hundreds of millions of images to train a model on them, CFG exists to deal with this problem, also Unstable Diffusion will just finetune a Stable Diffusion model, so you cannot simply change the licence to what you want.
- operator-name 4y agoBoorus [0] contain millions of images, manually labeled to a pretty high quality. Notably defusion models trained on booru datasets have had good success. This is not the only example of well curated image-tag pairs, especially in artistic circles. It's just that most of them are not CC. [0]: https://en.wiktionary.org/wiki/booru https://en.wiktionary.org/wiki/booru
- ritwikgupta 4y agoThis dataset is a massive failure when it comes to ethical research practices. LAION-5B openly indexed copyrighted data that it had no business collecting. They failed to go through an IRB when curating this data. The ethics review for this paper was a joke, where the ethics reviewer raises valid concerns and then discards their review because "if they don't publish it here, they'll publish it somewhere else anyways" [0]. LAION-5B has enabled some really cool technologies and a lot of promising startups. This work should have been carried out responsibly. [0] https://openreview.net/forum?id=M3Y74vmsMcY https://openreview.net/forum?id=M3Y74vmsMcY
- O__________O 4y agoWhat specifically are you claiming required a review board? Quick review of their site and the paper turns up nothing that commonly would be a topic that might merit such a review. Related FAQs: - https://laion.ai/faq/ https://laion.ai/faq/
- ritwikgupta 4y agoLAION-5B includes images of humans without their explicit consent. Images of people generally involve IRB/HSR. Additionally, almost any IRB will mention that if you’re using data derived from humans, you must go through IRB. LAION can say all they want that they’re not including images in their dataset. They include a script to download those URLs into images on disk. By being a company that’s not bound to decades of university ethics regulations, they are seemingly allowed to skirt what you learn on your first day as a researcher in academia. It may be legal, but it sure is not ethical.
- O__________O 4y agoPlease provide link to another academic publication agreeing with your claim that linking to online content is unethical without the subject’s explicit approval.
- nl 4y ago
- oth001 4y agoTerribly unethical using unlicensed images. They could/could've crowdsourced image gathering and labeling instead of stealing images.
- astrange 4y agoThis is like saying Google Image Search stole your image. (In fact it’s exactly the same; it’s allowed under the same laws and it respects robots.txt.)
- oth001 4y agoDoes Google.com allow anybody to instantly mimic an artist's style? Obviously AI laws haven't been put in place yet - it doesn't mean it's not unethical.
- astrange 4y agoIt's always been possible to imitate an artstyle. Nevertheless, they've never gotten IP protection - they're more like trade secrets. What's notable is "AI users are trying to copy an artist" != "AI has learned from an artist" != "AI has seen the artist's images in the first place". The most popular supposedly stolen-from artist Greg Rutkowski is not in StableDiffusion's training images, even though users are actively trying to copy him, it's a coincidence that it appears to work. Is that unethical? Also, AI laws (text and data mining exemptions) /have/ been put in place - to make this explicitly legal!
- oth001 4y agoImitate, not duplicate. Hours of work/talent vs. stealing an artist's work (see: unlicensed) and feeding it into a software program to spit out a slightly different version (complete with watermarks and all)...
- astrange 4y agoYou don't have much faith in an artist if you think you can "duplicate their style" by looking at 4-5 512x512 images of their work. > complete with watermarks and all That's not the AI "copying their watermarks", it's the AI learning "sometimes images have watermarks" and giving them some of its own.
- optymizer 4y agoSo.. the data set is licensed under Creative Commons, the source images all have their copyright (so how did they make it into the data set?) but what about images created with Stable Diffusion that uses this data set? Are those derived works? What license would they fall under?
- kmeisthax 4y agoThe license on the dataset only covers the collection of images and labels; which is not copyrightable in the US but copyrightable in the EU. The reason for this is the same reason why you can't copyright a phone book in the US but can in the EU[0]. The CC-BY license on the dataset only means you can avoid getting sued by the people who collected LAION-5B by attributing them. The copyright on the actual images and text labels is far more of a problem. Generally speaking, it is extremely infringing to collect a bunch of images or captions and redistribute them. Like, getting-to-the-heart-of-ownership, if-you-cant-sue-for-this-you-own-nothing kind of infringing. However, it's when you start talking about training ML systems on the dataset that things get interesting. In the EU, there's an explicit copyright exception for data mining that would apply to, say, training DALL-E or Stable Diffusion. But what about the US? Well, it's legal to crawl the web; and we specifically have Authors Guild v. Google where the Authors Guild lost in court trying to keep Google from scanning large numbers of books. AI researchers have sort of just taken this to mean "training AI is fair use". This is not court-tested, but it at least jives with some precedent, so I think it's OK to assume it's true. However, it means absolutely nothing for the people actually using the AI, because fair use is not transitive. If I take every YouTube video review of a movie and edit them down to just the movie clips being used, and then assemble them back together... I haven't somehow made a "fair use copy" of a movie that you can just share around. I've just made the most inefficient form of copyright infringement you can do with a computer. Likewise, if I train an AI on a movie, that can be fair use, but asking it to spit the movie back out is not. Now, keep in mind that some ML systems (such as Copilot) are very eager to reproduce their training set data. Sometimes in situations you wouldn't expect. These sorts of things are ticking time bombs for people who want to generate novel images, because the AI having trained on such a massive data set also gives you access to basically the whole data set. That's half of a US copyright infringement claim right there - the other half being substantial similarity, which basically is the "Corporate needs you to find the differences between these two pictures" meme. The only way to keep AI from infringing copyright is to make sure it never sees anything that could potentially be under copyright. [0] Strictly speaking, the EU has a separate concept of sui generis database ownership, but for this discussion we can treat it the same as copyright. If you're wondering why phone books aren't copyrightable, the term of art to search for is "sweat of the brow".
- deleted 4y ago[deleted]