2 ms·
Robots.txt is supposed to be public. Its purpose is to recommend not to visit some links (it's only a recommendation). And he did not visit the link itself, he
by realusername 12y ago
Robots.txt is supposed to be public. Its purpose is to recommend not to visit some links (it's only a recommendation). And he did not visit the link itself, he just read the comments (made for humans to read) in a public file, I don't think there is a problem here.