3 ms·
I like the idea of having a standard place to find this information. It's still for robots though isn't it? Why not include this information in robots.txt?
by will_critchlow 14y ago
I like the idea of having a standard place to find this information. It's still for robots though isn't it? Why not include this information in robots.txt?