3 ms·
Did you comment this in the wrong place? It doesn't seem to be a response to what I said. shagie brought up the possibility that somebody would have a robots.t
by akerl_ 4y ago
Did you comment this in the wrong place? It doesn't seem to be a response to what I said.
shagie brought up the possibility that somebody would have a robots.txt with only Google in it because when they wrote their robots.txt, they wanted to block poorly behaved / evil crawlers and Google's was well-behaved. But that doesn't make sense to me, because by definition poorly behaved / evil crawlers do not care about robots.txt.
- shagie 4y agoThe ones that weren't well behaved back then found themselves in a tarpit (which was also a 'what you do with spam email sites connecting on port 25'). https://en.wikipedia.org/wiki/Tarpit_(networking) https://en.wikipedia.org/wiki/Tarpit_(networking) The thing was that you'd send everything that was requesting faster than some rate limit to that tarpit unless it was google.
- rezonant 4y agoRiffing off what you wrote, not a rebuttal.