4 ms·

How are you this naive? Do you really think scrapers give a damn about your robots.txt?

• cpncrunch 11mo ago
  The legitimate ones do, which is what I was referring to. Obviously there are bastard ones as well.