5 ms·
Content you’re allowed and capable of scraping on the Internet is such a small amount of data, not sure why people are acting otherwise. Common crawl alone is
by oceanplexian 3y ago
Content you’re allowed and capable of scraping on the Internet is such a small amount of data, not sure why people are acting otherwise.
Common crawl alone is only a few hundred TB, I have more content than that on a NAS sitting in my office that I built for a few grand (Granted I’m a bit of a data hoarder). The fears that we have “used all the data” are incredibly unfounded.
- Eisenstein 3y ago> Facebook alone probably has more data than the entire dataset GPT4 was trained on and it’s all behind closed doors. Meta is happily training their own models with this data, so it isn't going to waste.
- bevekspldnw 3y agoNot Llama, they’ve been really clear about that. Especially with DMA cross-joining provisions and various privacy requirements it’s really hard for them, same for Google. However, Microsoft has been flying under the radar. If they gave all Hotmail and O365 data to OpenAI I’d not be surprised in the slightest.
- troq13 3y agoThe company that made a honeypot VPN to access competitor's traffic? They are definitively keeping their hands off their internal data, yes.
- deleted 3y ago[deleted]
- Dr_Birdbrain 3y agoI bet they are training their internal models on the data. Bet the real reason they are not training open source models on that data is because of fears of knowledge distillation, somebody else could distill LLaMa into other models. Once the data is in one AI, it can be in any AIs. This problem is of course exacerbated by open source models, but even closed models are not immune, as the Alpaca paper showed.
- sangnoir 3y ago> Content you’re allowed and capable of scraping on the Internet is such a small amount of data, not sure why people are acting otherwise YMMV depending on the value of "you" and your budget. If you're Google, Amazon or even lower tier companies like Comcast, Yahoo or OpenAI, you can scape a massive amount of data (ignoring the "allowed" here, because TFA is about OpenAI disregarding robots.txt)
- bevekspldnw 3y agoGonna say you’re way off there. Once you decompress common crawl and index it for FTS and put it on fast storage you’re in for some serious pain, and that’s before you even put it in your ML pipeline. Even refined web runs about 2TB once loaded into Postgres with TS vector columns, and that’s a substantially smaller dataset than common crawl. It’s not just a dumping a to of zip files on your NAS, it’s making the data responsive and usable.
- Dylan16807 3y agoHow important is full text search for training an LLM, compared to a pile of zip files with a gigabyte of text each?
- michaelt 3y agoMaybe not full full text search, but you'll generally want to remove the duplicates and suchlike.
- Dylan16807 3y agoI guess you want some fast extra storage for as long as it takes to run https://github.com/chatnoir-eu/chatnoir-copycat https://github.com/chatnoir-eu/chatnoir-copycat but that's a very temporary thing.
- bevekspldnw 3y agomy guess is mistral is so good for its size because they are doing some real precise pre ingestion selection and sorting. You need FTS to do that work.
- nicolas_17 3y ago"Content you're allowed to scrape from the internet" is MUCH smaller than what LLMs have actually scraped, but they don't care about copyright. > The fears that we have “used all the data” are incredibly unfounded. The problem isn't whether we used all the real data or not, the problem is that it becomes increasingly difficult to distinguish real data from previous LLM outputs.
- Dylan16807 3y ago> "Content you're allowed to scrape from the internet" is MUCH smaller than what LLMs have actually scraped, but they don't care about copyright. I don't know about that. If you scraped the same data and ran a search engine I think people would generally say you're fine. The copyright issue isn't the scraping step.