4 ms·
Literally Google Webmaster Guidelines say: "Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don't add much value
by thenextcorner 15y ago
Literally Google Webmaster Guidelines say:
"Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don't add much value for users coming from search engines."
So a couple of questions:
- What is of value for a user?
- Who is determining the value for the users in these cases?
As always, it's not always quite clear on what treatment you should use for search pages!
- dlitz 15y agoI'm pretty sure that having different search engines indexing each other would be bad. Don't cross the streams.
- thenextcorner 15y agoIf I recall correctly, crossing the streams did kill marshmallow man! In any case, I agree with you, search engine indexing search results would be bad, but the line is not that clear all the time! Some vertical search engine result pages are a great and relevant result from a user perspective on the question they are trying to solve.