4 ms·What if they update robots.txt?by maxerickson 2y agoWhat if they update robots.txt?toomuchtodo 2y agoRespecting robots.txt can be overridden.