3 ms·

Those cornercases can exist on sites that don't have a robots.txt and still have to be crawled correctly.