How exactly other websites “gave it all away”? Also examples you list are websites putting some explicit rule eg in their robots.txt or filtering web crawlers.

How exactly other websites “gave it all away”? Also examples you list are websites putting some explicit rule eg in their robots.txt or filtering web crawlers. This is all a reaction to existing situation, so Reddit for sure has been scrapped before Reddit realised what was happening.