3 ms·They say they obey robots.txt - isn’t that the easier way?by azinman2 8mo agoThey say they obey robots.txt - isn’t that the easier way?