3 ms·
I find this a bit hard to believe that they refuse to index it at all if the robots.txt doesn't exist. Would love to see more data or confirmation on this.
by johng 9mo ago
I find this a bit hard to believe that they refuse to index it at all if the robots.txt doesn't exist. Would love to see more data or confirmation on this.