3 ms·robots.txt is famously ignored by many ... agents ... out of pure spite. That's a human thing.by gerdesj 2y agorobots.txt is famously ignored by many ... agents ... out of pure spite. That's a human thing.