2 ms·only if the scraper/crawler actually pays attention to robots.txtby warrenm 4y agoonly if the scraper/crawler actually pays attention to robots.txtdutchbrit 4y agoWhich they ultimately should do when crawling a site. OpenAI does according to ChatGPT.warrenm 4y agoas a coworker of mine liked to say, "'should' is a funny word"