I agree with you, but I feel you are implying that we should not respect robots.txt any more: https://www.reddit.com/robots.txt https://www.reddit.com/robots.tx

I agree with you, but I feel you are implying that we should not respect robots.txt any more: https://www.reddit.com/robots.txt https://www.reddit.com/robots.txt The idea of ignoring robots.txt when it is abused for greed sounds okay, but it would be much better if the industry was in a spot where we respected each others' requests, and peers don't make requests that put legitimate, honest actors at a disadvantage compared to rouge crawlers