4 ms·

> aren’t there handling of this in stuff like robots.txt?... robots.txt only describes where a bot should go and where it shouldn't go, but nothing about what

by zeusk 8y ago

> aren’t there handling of this in stuff like robots.txt?... robots.txt only describes where a bot should go and where it shouldn't go, but nothing about what it should do with the content it discovers.

detaro 8y ago
robots meta-tags would be the place though.