3 ms·
Here's a concrete example of a site not worth archiving - back in the early days of the web (1994?) a friend of mine wrote a 4 function calculator. Each button
by dalke 16y ago
Here's a concrete example of a site not worth archiving - back in the early days of the web (1994?) a friend of mine wrote a 4 function calculator. Each button was a hyperlink, so "34-12" meant going to the pages "?", "?3", "?34", "?34-", "?34-1", and "?34-12". That's an infinite number of web pages which should not be indexed.
- thyrsus 16y agoI remember some anti-spam vigilante code that told legitimate bots to go away with robots.txt, told humans what the game was about, and attacked e-mail address scrapers by presenting an infinite number of fake e-mail addresses. "Click here for 1000 more e-mail addresses that will never bother a human!".
- alanh 16y agoI love it!
- rachelbythebay 16y agoNot all of them were fake. Some of them were spam traps which never appeared anywhere else. Some of those spam traps were also coded so you could track down when they were served to connect the spiders to the spewers. At least, that's the way I used to roll.
- smosher 16y agoAh very good! I was trying to think of a short example that wasn't contrived with this very behaviour.