3 ms·

They could add the ability to report sites that do this. That way there would be less sites to double crawl.