4 ms·

I'd rather see an open, search index whose URL's and scraping properties are shared with others. Then, small players, even researchers, can just rescrape the li

I'd rather see an open, search index whose URL's and scraping properties are shared with others. Then, small players, even researchers, can just rescrape the likely-good sites to build their own local stores. The whole Web filtered down to what is likely to be useful and safe (no malware) for a wide variety of people. If it needs to be paid for, then a commercial product that's priced by organization size. Given to researchers for free if their outputs are non-commercial or permissive licensed. Discounted otherwise. I usually start with how Windows is priced for personal or server use to be profitable and widely accessible. That would let people re-create data sets like RefinedWeb without violating copyright law. You'd still have to consider terms of service, contract law, etc. We have stronger, legal defenses of scraping for internal use, especially non-commercial. Knocking out copyright issues would be a huge help.

latexr 2y ago
https://commoncrawl.org/ https://commoncrawl.org/
- nickpsecurity 2y ago
  Common Crawl isn't open source. It's license has many, specific restrictions. Some are often interpreted in specific ways that could be censoree due to politics. I'm not sure what CC organizations interpretations are, though. I did think they'd be the best at making the URL part of what I described. Their CC dumps are copyright infringement (file sharing) but links probably would be legal. It would need to be released under an open-source license without their extra terms. Alternatively, a no-terms license given to specific, paying parties for internal use. Alternatively, released under non-commercial use with low, recurring prices for (a) commercial use and (b) regular updates on link Metadata. Paid alternatives might be good for their funding. I can't use them right now, though, because I can't guarantee all my or customers' uses meet all likely interpretations of their terms. I'm not even sure how to put best effort in that. I'd rather them just publish their metadata under Apache license or something.
ricardo81 2y ago
I think there's a few problems with that kind of setup. Quick thoughts: - Content creators have less discretion on who to allow/block crawling for when there's a middleman index (probably doesn't matter so much now given the flagrant use of content for AI) - Content recency. The data sizes can get quite huge, and certain pages require updates more often than others so who gets to decide (one user of the index may be interested in a different set of pages vs another) - Centralised content on the likes of Reddit, who are already aggressively blocking most bots from crawling their content. You'd have to crawl many pages per day (and quite likely end up getting blocked) as generally only a handful of bots get favourable treatment to crawl sites more aggressively.
- immibis 2y ago
  So you can't block search engines other than your favorite - that's a positive. If you want to block one you have to block them all. Recency is not a problem any more than it's a problem for one individual search crawler
- mdaniel 2y ago
  > - Content recency. The data sizes can get quite huge, and certain pages require updates more often than others I have always imagined that having an open crawl corpus aligns closely with the goals of the Internet Archive, where one could already strictly speaking submit updates to with second-level precision based on their URL slugs. The bad news is that with any such common corpus it would actually worsen their bandwidth bill since I would highly suspect that a corpus would be read from much more than it would ingest (e.g. snap https://news.ycombinator.com/item?id=43318384 https://news.ycombinator.com/item?id=43318384 once but then every downstream corpus consumer would read from IA n times) While typing this out, I actually wonder if the big search players don't maintain "page diffs" in their index such that loading https://news.ycombinator.com/item?id=43311573 https://news.ycombinator.com/item?id=43311573 an hour ago and then loading https://news.ycombinator.com/item?id=43311573 https://news.ycombinator.com/item?id=43311573 now stores only the new content > so who gets to decide (one user of the index may be interested in a different set of pages vs another) Surely that's a solved problem in that a common corpus would ingest updates to all frontier pages that it knows about, and exponentially back-off as it finds less and less updates. I don't think the CommonCrawl.org cited by the sibling comment is selective about updates, and unquestionably IA does not: they accept snapshot requests from anyone for what I presume is any URL

latexr 2y ago

https://commoncrawl.org/ https://commoncrawl.org/

nickpsecurity 2y ago
Common Crawl isn't open source. It's license has many, specific restrictions. Some are often interpreted in specific ways that could be censoree due to politics. I'm not sure what CC organizations interpretations are, though. I did think they'd be the best at making the URL part of what I described. Their CC dumps are copyright infringement (file sharing) but links probably would be legal. It would need to be released under an open-source license without their extra terms. Alternatively, a no-terms license given to specific, paying parties for internal use. Alternatively, released under non-commercial use with low, recurring prices for (a) commercial use and (b) regular updates on link Metadata. Paid alternatives might be good for their funding. I can't use them right now, though, because I can't guarantee all my or customers' uses meet all likely interpretations of their terms. I'm not even sure how to put best effort in that. I'd rather them just publish their metadata under Apache license or something.

ricardo81 2y ago

I think there's a few problems with that kind of setup. Quick thoughts: - Content creators have less discretion on who to allow/block crawling for when there's a middleman index (probably doesn't matter so much now given the flagrant use of content for AI) - Content recency. The data sizes can get quite huge, and certain pages require updates more often than others so who gets to decide (one user of the index may be interested in a different set of pages vs another) - Centralised content on the likes of Reddit, who are already aggressively blocking most bots from crawling their content. You'd have to crawl many pages per day (and quite likely end up getting blocked) as generally only a handful of bots get favourable treatment to crawl sites more aggressively.

immibis 2y ago
So you can't block search engines other than your favorite - that's a positive. If you want to block one you have to block them all. Recency is not a problem any more than it's a problem for one individual search crawler
mdaniel 2y ago
> - Content recency. The data sizes can get quite huge, and certain pages require updates more often than others I have always imagined that having an open crawl corpus aligns closely with the goals of the Internet Archive, where one could already strictly speaking submit updates to with second-level precision based on their URL slugs. The bad news is that with any such common corpus it would actually worsen their bandwidth bill since I would highly suspect that a corpus would be read from much more than it would ingest (e.g. snap https://news.ycombinator.com/item?id=43318384 https://news.ycombinator.com/item?id=43318384 once but then every downstream corpus consumer would read from IA n times) While typing this out, I actually wonder if the big search players don't maintain "page diffs" in their index such that loading https://news.ycombinator.com/item?id=43311573 https://news.ycombinator.com/item?id=43311573 an hour ago and then loading https://news.ycombinator.com/item?id=43311573 https://news.ycombinator.com/item?id=43311573 now stores only the new content > so who gets to decide (one user of the index may be interested in a different set of pages vs another) Surely that's a solved problem in that a common corpus would ingest updates to all frontier pages that it knows about, and exponentially back-off as it finds less and less updates. I don't think the CommonCrawl.org cited by the sibling comment is selective about updates, and unquestionably IA does not: they accept snapshot requests from anyone for what I presume is any URL