This is helpful, thank you. You mean to say that spambots don't respect robots.txt?? Good to think about. I'll check these out and see what they've done (if m

This is helpful, thank you. You mean to say that spambots don't respect robots.txt?? Good to think about. I'll check these out and see what they've done (if much) to mitigate that issue. I'd imagine that would need to be front and center, unless they're focused more on internal services.