You really didn't think about this. Search engine crawlers are specifically designed to ignore sites that have the right setups. In other words: they try to be

You really didn't think about this. Search engine crawlers are specifically designed to ignore sites that have the right setups. In other words: they try to be good citizens of the web. Data scrapers on the other hand are known to steal others IP and use it to jump start their own services. When you do that you're using all the bandwidth of the site while devaluing everything they've built. They will have invested significant time and resources building that IP. But data scrapers think that just because its on a public website it means they can leech it and do what they like. No, there is such a thing as copy right and respecting authors. Fuck anyone who says otherwise. If you want to be a little kiddiot and steal content you're free to do so. But you're just undermining the work of everyone you steal from. Making it less and less viable for them to make more of it in the future. I stand by what I said. Data miners are fucking parasites and the web would be better off without them.