4 ms·
Yep, it's defined at https://www.robotstxt.org/ https://www.robotstxt.org/ Looks like Google is trying to turn it into an RFC though.
by banana_giraffe 6y ago
Yep, it's defined at https://www.robotstxt.org/ https://www.robotstxt.org/
Looks like Google is trying to turn it into an RFC though.