3 ms·Considering the effort that went into it. I am pretty sure crawling robot.txt links was their P1 requirement.by rk06 9y agoConsidering the effort that went into it. I am pretty sure crawling robot.txt links was their P1 requirement.