3 ms·And many bots don't take any notice of robots.txt anyway.by simonblack 3y agoAnd many bots don't take any notice of robots.txt anyway.