3 ms·Google and others, with legitimate reasons obey robots.txtby desireco42 10y agoGoogle and others, with legitimate reasons obey robots.txt