> I wish the IA were little more aggressive about not retroactively applying robots.txt rules on archived content. They've stopped doing that. They now ignore

> I wish the IA were little more aggressive about not retroactively applying robots.txt rules on archived content. They've stopped doing that. They now ignore robots.txt completely and you have to email them to stop them.