4 ms·
Maybe crawling should be a common utility, with the ranking being left to individual competition, similar to fiber lines owned by utilities and then companies b
by ericd 8y ago
Maybe crawling should be a common utility, with the ranking being left to individual competition, similar to fiber lines owned by utilities and then companies being free to lease those and deal with end customers. As it is, many websites are hostile to new crawlers that don't already have lots of traffic to bring them in exchange, so it's difficult to offer an alternative to Google, even if you can get over the high minimum bar of being able to crawl a significant fraction of the web.
- bduerst 8y agoIsn't that just creating a duplicate internet then - one that taxpayers have to pay to host instead? That creates a ton of issues around being forced to pay for hosting other people's speech. For example, I don't want to pay to host anti-vaxxer site information but because of the first amendment, this public service can't remove it.
- ericd 8y agoWhy would you care about the cost to store a copy of sites you disagree with? It's a crawl, it's not the government making value judgements one way or the other. I'm sure the library of congress also contains material you disagree with. I don't think it would count as a duplicate internet any more than archive.org would.
- bduerst 8y agoThe library of congress is curated by Selection officers for material value, the internet is not.
- deleted 8y ago[deleted]
- deleted 8y ago[deleted]
- ericd 8y agoI didn't mean to say that the library of congress was indiscriminate, just that it likely also contains material that you disagree with. And that that doesn't really matter, because it's not the purpose of the repository to contain only non-controversial material.
- bduerst 8y agoWithout an example, I can't agree with you, and no - the point is that the library of congress is curated content, the internet is not.
- ericd 8y agoOld books from the 1800s with very outdated views on race, sex, etc would be an obvious example. Edit: and why is curation relevant to the question of whether there should be a common crawl that can be accessed by all, a publicly hosted mirror of the web that can be used to create competing search engines?
- bduerst 8y agoI really don't care about old books from the 1800s.