2 ms·

I'd be OK with an open reciprocal crawling network for non-personal/private pages as it would be a distributed force against walled gardens. I'm very against t

• mdaniel 1y ago
  I ran an extension that automatically submitted pages to the Internet Archive as I browsed them, but managing the allowlist/denylist turned into a major hassle, so I eventually just installed the extension into a "public browsing" profile, but as is often the case it turned into "I don't feel like switching to that profile" and it fell by the wayside But, in the same vein as your comment, I have long wished for Common Crawl to really lean into their mission, and not just publish monthly snaps of whatever their bots can see but do what you said and accept .har or .warc files from anyone and serve the ... hourly? ... .warc via Bittorrent