3 ms·Do any scrapers actually pay attention to robots.txt?by paulie_a 7y agoDo any scrapers actually pay attention to robots.txt?