4 ms·

Use a user-agent containing a URL to find out who and what you are, and honor my robots.txt. Having a panel for webmasters along with that would be fine.

• pranade 13y ago
  Great suggestion... thanks for this one. We're putting this on our list
  • thatthatis 13y ago
    Please tell me that the robots.txt suggestion is something that you're already doing and the user agent part is whats going on the list.
    • garyjob 13y ago
      You could try doing IP address rotation by rotation EC2 instances or some other cloud services. I wrote a library for that. https://github.com/KrakeIO/resque-my-aws https://github.com/KrakeIO/resque-my-aws
      • Kudos 13y ago
        That has nothing to do with what's being discussed on this thread.