3 ms·
MMC4: An open, billion-scale corpus of images interleaved with text
- andyjohnson0 4y agoWhats the best source for a directory/list of these training data corpuses?
- ftxbro 4y ago"We introduce mmc4, a corpus of 585M images interleaved in 43B English tokens from the popular c4 dataset." This training corpus mmc4 is the one used by OpenFlamingo (https://laion.ai/blog/open-flamingo/ https://laion.ai/blog/open-flamingo/). Something caught my eye when I read the pdf. They are using GPT-4 to name their word clusters: "Topic names are generated by GPT-4 conditioned on the top 20 words for each topic, prompted by a request for a short 1-2 word summary." I feel like LLMs will be used increasingly for this purpose; whenever someone has to make a mostly arbitrary decision like the kind that would make 'bike shedding arguments', they can delegate it to the LLM. The point isn't that the LLM is necessarily better or objective, but rather that it's a kind of 'schelling point' for agreeing on things that don't matter. For example I could imagine a super annoying peer reviewer (or even co-author) ask why you used the name 'celebrations' as a topic name associated to the set of words 'fun, wedding, beautiful, christmas, happy, card, birthday, gift, blog, perfect' like in the pdf, and why not use some other word. Instead of having some meaningless back and forth 'bike shedding' discussion with the reviewer over email, you can just say you used GPT-4 and move on to more important things.
- jmmcd 4y agoGreat! Now the reviewer will say you should've used GPT4.2, the version released on 29th August, not the version released on 22nd August.
- LawTalkingGuy 4y agoLol, that's what a reviewer running on Llama13b would say!
- taneq 4y agocalls out to nearby microphone "hey bikeshed, what's the best llm and revision for this debate?"
- seydor 4y agoBut then how can their license allow commercial use?
- angrais 4y agoAll this data is taken from common crawl, a web crawler that clones the web. How can companies such as OpenAI use that data when no licenses can be identified for most web pages or the associated images? In other words, the user/owner of the content has not agreed for it to be used in these ways. What's people's experiences with training models using such data? Is there any way to identify if my photos appear in common crawl or are used in such datasets?
- flangola7 4y agoSame way Copilot uses GitHub code and Midjourney uses scraped art images.
- brookst 4y agoKeep on mind that humans may also be learning from your work and emulating aspects of it.
- kolinko 4y agoYou have robots.txt standard that you can use to exclude crawling.
- gleenn 4y agoI believe there is an argument to be made that it is a derivative work. If you read a copyrighted book verbatim, that's an infringement, but if you summarize a book, that's fair-use. I don't think almost any of this has gone through the courts so there is a lot of legal peril and speculation though, obviously this is all super hot and pushing the boundaries so I'm sure we'll get clarification sooner rather than later.