3 ms·
> There's a real conflict here. I want Google and Bing and DuckDuckGo to crawl my site and send me new readers. But I don't want everyone else strip-mining it.
by andai 3mo ago
> There's a real conflict here. I want Google and Bing and DuckDuckGo to crawl my site and send me new readers. But I don't want everyone else strip-mining it.
Kinda sounds like we're missing a peer to peer network here.
Instead of downloading the same data over and over again we can just download it once and then share it.
Wouldn't that be better for everyone involved?
It would also function as a distributed WayBack Machine, in case anything ever happens to the Internet Archive. (Which I think is desperately needed, bot apocalypse aside.)
- amazingamazing 3mo agoThe problem there is trust
- econ 3mo agoYou can validate by repeating the work and sign your content. The domain name system is now just a rent seeking scheme. It was great as a temp solution but over time it has deleted more content than preserved. I might in theory be billed for having a country name but I don't pay for a city, street name, house number or postal code. Online you should be able to move your widget shop to widget street. Can bill people who want to live on real estate street or used car street and/or set some requirements.
- NicuCalcea 3mo agoCommon Crawl? https://commoncrawl.org/ https://commoncrawl.org/
- ccgreg 3mo agoThe author blocked CCBot even though CCBot isn't part of the high traffic problem -- apparently he trusted Cloudflare labeling us as an "AI Bot".
- inigyou 3mo agoAnyway, Google doesn't send traffic to your site any more. Only important sites and obvious scams seem to get indexed.
- potamic 3mo agoHave you checked out IPFS?