5 ms·All major search engines follow robots.txtby occamrazor 6y agoAll major search engines follow robots.txt