4 ms·

Please tell me that the robots.txt suggestion is something that you're already doing and the user agent part is whats going on the list.

• garyjob 13y ago
  You could try doing IP address rotation by rotation EC2 instances or some other cloud services. I wrote a library for that. https://github.com/KrakeIO/resque-my-aws https://github.com/KrakeIO/resque-my-aws
  • Kudos 13y ago
    That has nothing to do with what's being discussed on this thread.