4 ms·

Is there a good reason that reddit has banned 80legs crawling?

• jdrock 15y ago
  True story: We crawled them a while back (before they expanded their engineering team) and because of our distributed system, "alarms" were going off. Rather then take time to tell their system we were not a DDOS attack, they put us in robots.txt. I imagine their small team had other stuff to work on.
  • xtacy 15y ago
    So, 80legs caused more traffic volume to reddit than Google crawler? That seems interesting; why would it be the case?
    • jdrock 15y ago
      Not necessarily more traffic, though that's possible. More likely it was the fact that it was coming from multiple IP addresses.