3 ms·

Great suggestion... thanks for this one. We're putting this on our list

• thatthatis 13y ago
  Please tell me that the robots.txt suggestion is something that you're already doing and the user agent part is whats going on the list.
  • garyjob 13y ago
    You could try doing IP address rotation by rotation EC2 instances or some other cloud services. I wrote a library for that. https://github.com/KrakeIO/resque-my-aws https://github.com/KrakeIO/resque-my-aws
    • Kudos 13y ago
      That has nothing to do with what's being discussed on this thread.