3 ms·some people don't want to be scrapped at all, and most bots ignore `robots.txt`by tyre 26d agosome people don't want to be scrapped at all, and most bots ignore `robots.txt`