> So that is how they block a people's search/crawler. It seems less that was the intent and more they blacklist IPs with behavior they find annoying. They're

> So that is how they block a people's search/crawler. It seems less that was the intent and more they blacklist IPs with behavior they find annoying. They're super general lists. > They also use block lists to add every single TOR node This annoyed the crap out of me. The stupid Dan list guy made it as easy as he could to lump low risk bridges in with high risk exit nodes.