12 ms·I mean if they want to "wriggle out" of robots.txt they can just ignore it. It's entirely voluntary.by mvdtnz 8mo agoI mean if they want to "wriggle out" of robots.txt they can just ignore it. It's entirely voluntary.