4 ms·
Who exactly is running all these scrapers? There are, what, maybe 15 major AI labs, if that? And none of them are smart enough to realize they could just `git
by Wowfunhappy 1mo ago
Who exactly is running all these scrapers? There are, what, maybe 15 major AI labs, if that?
And none of them are smart enough to realize they could just `git clone` all the content and use it offline?
- strix_varius 1mo agoThere are many more labs than that, and humans aren't designing unique scraping processes per domain.
- micah_chatt 1mo agoIf you do the math (also a common system design interview question for an AI lab), its actually only ~3PB (compress to 1PB, ~$22,000/mo in S3) and a few thousand/mo in compute over less than 4 months to index the entire internet for pretraining purposes. At that price point, its actually very affordable to many thousands of organizations to get their own copy. I would expect the major labs to special case kernel.org similarly to other sites like Wikipedia, but not the majority of scrapers
- Wowfunhappy 1mo ago> its actually only ~3PB (compress to 1PB, ~$22,000/mo in S3) and a few thousand/mo in compute over less than 4 months to index the entire internet for pretraining purposes. That is super interesting, thank you! > At that price point, its actually very affordable to many thousands of organizations to get their own copy. I'm still confused as to who is actually doing it though! Maybe it's affordable to scrape and store, but training a competitive AI model is going to cost much more, right?
- DarmokTanagra 1mo agothree guesses what geographic regions they are based in.
- GoblinSlayer 1mo agoI suppose proxies provide only http request interface.