Every website operator should be required to learn _rate-limiting_ as well as how to create a robots.txt file What matters is rate not "user-agent" User-agent

Every website operator should be required to learn _rate-limiting_ as well as how to create a robots.txt file What matters is rate not "user-agent" User-agent can be fake but rate is real