3 ms·

There are enough malicious bots that do follow robots.txt to make it still an important option for most sites.