4 ms·
A simple robots.txt is all that was needed. But that wasn’t used by News Corp, so that suggests to me that they were just trying to ride on Google’s coat tails
by alfiedotwtf 6y ago
A simple robots.txt is all that was needed. But that wasn’t used by News Corp, so that suggests to me that they were just trying to ride on Google’s coat tails