4 ms·
Not like anyone writing a spider has to obey the robots.txt file.
by antipax 17y ago
Not like anyone writing a spider has to obey the robots.txt file.
- DrJokepu 17y agoI smell a business opportunity here! I could develop a Digital Robots.txt Management platform that will be built into all server operating systems and would enforce that no spiders are allowed to disregards robots.txt. I'm a genius! Oh, wait...
- eli 17y agoSure, and their IP will be blocked by Google some time around the 10th request.
- jacquesm 17y agoNext time you're looking at a PC infected with malware have a look at the network traffic using a sniffer, chances are pretty good that you'll see searches to google for the weirdest of terms. Apparently this is to get around the limitation that you mention. I'm assuming the results of such searches will be 'mailed home' through some kind of dead-drop.
- eli 17y agoSure, but responding to the parent, that's why people don't crawl Google, not the Robots.txt. And incidentally, some of those searches are to find forms that it can stuff links into. My sites are constantly getting hit by botnets searching for Drupal comment forms. Luckily it uses a quirky URL format that's easy for mod_security to block.