3 ms·
I am confused. Where does robots.txt come into play here? I always thought that if url is allowed in robots.txt it's fair game to scrape.
by zaro 7y ago
I am confused. Where does robots.txt come into play here?
I always thought that if url is allowed in robots.txt it's fair game to scrape.