4 ms·
This is literally the point of robots.txt. It was created to allow site owners to configure how and which parts of their website can be scraped by what bot, and
by sofixa 9mo ago
This is literally the point of robots.txt. It was created to allow site owners to configure how and which parts of their website can be scraped by what bot, and all the "decent" ones (Google, Bing) respect it.