3 ms·
Crawlers ignoring robots.txt is abusive. That they then start scanning all docs for commented urls just adds to the pile of scummy behaviour.
by michael1999 11mo ago
Crawlers ignoring robots.txt is abusive. That they then start scanning all docs for commented urls just adds to the pile of scummy behaviour.
- tveyben 11mo agoHuman behavior is interesting - me, me, me…