So it's more out of ignorance than maliciousness. That makes sense. Odds are the more troublesome scrapers are probably ignoring robots.txt anyway, so I wonder

So it's more out of ignorance than maliciousness. That makes sense. Odds are the more troublesome scrapers are probably ignoring robots.txt anyway, so I wonder how useful it is.