6 ms·
The end state of training on web text has always been an ouroboros - primarily because of adtech incentives to produce low quality content at scale to capture m
by bevekspldnw 3y ago
The end state of training on web text has always been an ouroboros - primarily because of adtech incentives to produce low quality content at scale to capture micro pennies.
The irony of the whole thing is brutal.
- ezekg 3y ago> The end state of training on web text has always been an ouroboros And when other mediums have been saturated with AI? Books, music, radio, podcasts, movies -- what then? Do we need a (curated?) unadulterated stockpile of human content to avoid the enshittification of everything?
- weregiraffe 3y ago>Do we need a (curated?) unadulterated stockpile of human content to avoid the enshittification of everything? Either that, or a human level AI.
- ethanbond 3y agoWell no, we need billions of human-level AIs who are experiencing a world as rich and various as the world that the billions of humans inhabit.
- ben_w 3y agoOnce we've got the first, making a billion is easy. That said… are content creators collectively (all media, film and books as well as web) a thin tail or a fat tail? I could easily believe most of the actual culture comes from 10k-100k people today, even if there's, IDK, ten million YouTubers or something (I have a YouTube channel, something like 14 k views over 14 years, this isn't "culturally relevant" scale, and even if it had been most of those views are for algorithmically generated music from 2010 that's a literal Markov chain).
- xipho 3y agoMaybe we could print out knowledge on dead trees and store them somewhere, perhaps make it publically available? (stolen joke, not mine).
- berniedurfee 3y agoYahoo.com will rise from the ashes.
- bevekspldnw 3y agoI mean, you’re not wrong. I’ve been building some unrelated web search tech and have considered just indexing all the sites I can about and making my own “non shit” search engine. Which really isn’t too hard if you want to do say, 10-50 sites. You can fit that on one 4TB nvme drive on a local workstation. I’m trying to work on monetization for my product now. The “personal Google” idea is really just an accidental byproduct of solving a much harder task. Not sure if people would pay for that alone.
- nicolas_17 3y agohttps://en.wikipedia.org/wiki/Low_background_steel https://en.wikipedia.org/wiki/Low_background_steel but for web content.
- oceanplexian 3y agoContent you’re allowed and capable of scraping on the Internet is such a small amount of data, not sure why people are acting otherwise. Common crawl alone is only a few hundred TB, I have more content than that on a NAS sitting in my office that I built for a few grand (Granted I’m a bit of a data hoarder). The fears that we have “used all the data” are incredibly unfounded.
- Eisenstein 3y ago> Facebook alone probably has more data than the entire dataset GPT4 was trained on and it’s all behind closed doors. Meta is happily training their own models with this data, so it isn't going to waste.
- bevekspldnw 3y agoNot Llama, they’ve been really clear about that. Especially with DMA cross-joining provisions and various privacy requirements it’s really hard for them, same for Google. However, Microsoft has been flying under the radar. If they gave all Hotmail and O365 data to OpenAI I’d not be surprised in the slightest.
- troq13 3y agoThe company that made a honeypot VPN to access competitor's traffic? They are definitively keeping their hands off their internal data, yes.
- deleted 3y ago[deleted]
- Dr_Birdbrain 3y agoI bet they are training their internal models on the data. Bet the real reason they are not training open source models on that data is because of fears of knowledge distillation, somebody else could distill LLaMa into other models. Once the data is in one AI, it can be in any AIs. This problem is of course exacerbated by open source models, but even closed models are not immune, as the Alpaca paper showed.