3 ms·
Good question. robots.txt was designed for an age before ajax and the proliferation of plugins/toolbars/etc. It's a valid shortcoming you've identified, imho.
by user24 16y ago
Good question. robots.txt was designed for an age before ajax and the proliferation of plugins/toolbars/etc. It's a valid shortcoming you've identified, imho.