3 ms·So the news sites can add a single "*" to their robots.txt. Problem solved.by brokenmachine 6y agoSo the news sites can add a single "*" to their robots.txt. Problem solved.