2 ms·Respecting robots.txt is probably the best plan.by quesera 14y agoRespecting robots.txt is probably the best plan.