> does not respect robots.txt research shows. It would be nice then for the investigators to help people with the identifying markers for such crawlers. Apart

> does not respect robots.txt research shows. It would be nice then for the investigators to help people with the identifying markers for such crawlers. Apart from a mention of darkvisitors, which it seems is a paid service to "Block agents who try to ignore your robots.txt" I'm not sure how much that could be trusted given their business model also.