The reason the request rate was at 4 was that at some point your site responded to 4 req/sec without problems. User-agent is the accepted and standardized way

The reason the request rate was at 4 was that at some point your site responded to 4 req/sec without problems. User-agent is the accepted and standardized way to communicate with bots. Getting angry at us for following robots.txt is like getting angry at someone for showing you her ID when she goes through airport security. From our perspective, your house put up a sign that said "Hey everyone come in 4 at a time".