4 ms·
The irony of the situation is the crawlers specifically allowed are most likely the only ones that bother to abide by robots.txt in the first place. It'll be in
by meritt 13y ago
The irony of the situation is the crawlers specifically allowed are most likely the only ones that bother to abide by robots.txt in the first place. It'll be interesting to see if the change has any impact whatsoever.