3 ms·Well-behaved agents will obey robots.txt and not fall into the trap.by phoronixrly 7mo agoWell-behaved agents will obey robots.txt and not fall into the trap.