3 ms·
IMHO the idea of accepting some bots (e.g. Google Bot) while not accepting others in robots.txt (e .g. a new Google business killer bot) clashes with Internet n
by wslh 5y ago
IMHO the idea of accepting some bots (e.g. Google Bot) while not accepting others in robots.txt (e
.g. a new Google business killer bot) clashes with Internet neutrality.