2 ms·
Well, there never was any law that required robots.txt to be honored. Big players like Google do, but I am not aware of any consequences if they wouldnt (of cou
by littlecranky67 2y ago
Well, there never was any law that required robots.txt to be honored. Big players like Google do, but I am not aware of any consequences if they wouldnt (of course UNTIL it is then regulated).