4 ms·
In defence of search engines, when content disappears from the web, it disappears from the search index within months. And robots.txt has been a thing since for
by lstamour 5y ago
In defence of search engines, when content disappears from the web, it disappears from the search index within months. And robots.txt has been a thing since forever.
But I agree with the overall point, that archiving is distinct from indexing, and we need better mechanisms than copyright to take down content given many services terms allow content to be kept forever and/or make it hard to delete content uploaded by others. This might come down to free speech vs privacy, though.
- floatingatoll 5y agoIf I ever again post a website, I’m going to post it with basic auth, so that search engines won’t index it to begin with. Exposure doesn’t pay the bills, after all.
- Sanzig 5y ago> And robots.txt has been a thing since forever. Unfortunately, both archive.org and the affiliated Archive Team now intentionally ignore robots.txt files. Archive Team, in particular, wrote a very irritating and condescending screed called "ROBOTS.TXT IS A SUICIDE NOTE" [1]. Apparently, they don't understand that not everyone wants their stuff backed up forever and they treat anyone who disagrees with them as a moron. And that manifesto wasn't written by some random team member, it was from Jason Scott himself. I know that Archive Team isn't officially part of IA, but they are tightly integrated with their archiving operations. It's a bad look. [1] http://wiki.archiveteam.org/index.php?title=Robots.txt http://wiki.archiveteam.org/index.php?title=Robots.txt