3 ms·robots.txt is relevant and effective, as is my DNT header.by elpocko 3y agorobots.txt is relevant and effective, as is my DNT header.