3 ms·
Not really. robots.txt has always been based on a good faith. All the reputable companies abide by it, but I don't doubt for a second they'd stop reading it i
by jlarocco 15y ago
Not really.
robots.txt has always been based on a good faith.
All the reputable companies abide by it, but I don't doubt for a second they'd stop reading it if it started affecting their bottom line.