A missing robots.txt has always meant "allow all", so it would be really weird for this to be the cause. Robots.txt is primarily for disallowing (or really, dis

A missing robots.txt has always meant "allow all", so it would be really weird for this to be the cause. Robots.txt is primarily for disallowing (or really, discouraging) crawlers from crawling certain paths.