We had never seen this before. I understand their crawler has to be very aggressive, but it seems to me like the indexing is broken, at least in this case. Ima

We had never seen this before. I understand their crawler has to be very aggressive, but it seems to me like the indexing is broken, at least in this case. Imagine if someone gets all our pages and puts them somewhere else. Even without links to them, they could take over us. Thanks for the idea of the robots.txt. Since the admin site is really for testing and eventually released, it's some work to make the robots.txt dynamic. May do it soon just to make sure.