3 ms·
I agree that there needs to be better control over such things. Whatever comes from this, it also needs to avoid the fundamental problem with robots.txt: it's a
by JohnFen 3y ago
I agree that there needs to be better control over such things. Whatever comes from this, it also needs to avoid the fundamental problem with robots.txt: it's advisory only, and bots can freely ignore it. And many do.
A real solution has to have an effectiveness greater than just asking nicely and hoping that people are honorable.