4 ms·

I have no association with this person and just heard about this. But obviously those are search term result pages and not "autogenerated pages" in the usual se

I have no association with this person and just heard about this. But obviously those are search term result pages and not "autogenerated pages" in the usual sense. If Google is going to apply such a standard uniformly, they'd have to ban all sites that have search functions. Autogenerated in the bad sense is link farms like fakesite.com/buy_drugwiththisname_now.html, with drugwiththisname replaced with 500,000 different possibilities, and all generating pages linked to each other, which also have incoming links from ten billion forum comments across the web where a bot has signed up en masse and posted spam. Now, perhaps this guy is posting spam to his site in places which makes it valid to declare this a link farm and kill it. Looking at the site now, being able to search amazon by exact price is a pretty neat function and is totally different from a link farm. I think killing his site for having a search function is pretty unreasonable. However, you are a private for-profit company, so you can do as you please obviously.

chime 15y ago
> Autogenerated in the bad sense is link farms like fakesite.com/buy_drugwiththisname_now.html, with drugwiththisname replaced with 500,000 different possibilities Actually there is no difference between search page result sites like the OP and what you said. Having a URL that ends in .html does not mean there is a static html file on the server. For examples, nextag and pricegrabber have URLs like blah.com/digital/Canon-EOS-7D-SLR-Digital-Body/m739295014.html. Scroll around these sites and you'll see they're anything but static. Each page is simply a result of a query. Whether the database is local (like nextag) or remote (like Amazon API query like OP) is inconsequential. Personally, I would be VERY happy if Google hid/ignored these kind of search-query sites. I have not once found any of these sites useful. Problem is the large gray area in which these sites operate and what I find useless, someone else might find useful.
- StrawberryFrog 15y ago
  > Actually there is no difference between search page result sites like the OP and what you said Uh, yes there is. http://www.somesite.com/index.html?q=foo http://www.somesite.com/index.html?q=foo is obviously a search (right down to the choice if q for query), but http://www.somesite.com/buy_foo_now.html http://www.somesite.com/buy_foo_now.html does not look like one. You might say that technically they could have the same back-end, but the web is flexible - just about any URL scheme could resolve to the same back-end. The difference is that the first is honest and the second one goes out of its way to hide what it does.
  - ceejayoz 15y ago
    > Uh, yes there is. http://www.somesite.com/index.html?q=foo http://www.somesite.com/index.html?q=foo is obviously a search Obviously? Drupal's default non-rewritten URLs are index.php?q=foo, where foo is the page path. `q` can and does stand for more than one thing.
- robryan 15y ago
  Would be easy for the google bot to work this out to, replace the URL part that looks like a dynamic variable with different values and see if it returns a page for them all like a search engine would. Surprised it doesn't appear to do this.
joshu 15y ago
Search results are also not supposed to be indexed. I believe this is actually in the guidelines.
- Matt_Cutts 15y ago
  Yup. Search for [quality guidelines] and they're at http://www.google.com/support/webmasters/bin/answer.py?answer=35769 http://www.google.com/support/webmasters/bin/answer.py?answe... . The relevant part is "Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don't add much value for users coming from search engines."
  - j_col 15y ago
    Matt, I tried that recently with one of my own sites, and after a few days I got a warning in my Google Webmaster Tools dashboard that the bot could not access my /search URL because it was blocked in my robots.txt, and I should take action to correct it. So I then unlocked my /search URL, which is in violation of the above guideline, but made the error in Google Webmaster Tools go away. These conflicting messages from Google are very confusing!
    - vanessafox 15y ago
      Hi j_col- In that case, Google Webmaster tools is not actually reporting an error. That's a report to show you what URLs Google tried to crawl but couldn't (due to being blocked) so you can review it and ensure that you are not accidentally blocking URLs that you want to have indexed. I agree that it's confusing in that the report is in the "crawl errors" section. (I built Google webmaster tools so this confusion is entirely my fault; but I don't work at Google anymore so sadly I can't fix this.)
      - j_col 15y ago
        Thanks for the response, I will block my /search URL once more via robots.txt and will ignore the warnings in the Webmaster Tools.

chime 15y ago

> Autogenerated in the bad sense is link farms like fakesite.com/buy_drugwiththisname_now.html, with drugwiththisname replaced with 500,000 different possibilities Actually there is no difference between search page result sites like the OP and what you said. Having a URL that ends in .html does not mean there is a static html file on the server. For examples, nextag and pricegrabber have URLs like blah.com/digital/Canon-EOS-7D-SLR-Digital-Body/m739295014.html. Scroll around these sites and you'll see they're anything but static. Each page is simply a result of a query. Whether the database is local (like nextag) or remote (like Amazon API query like OP) is inconsequential. Personally, I would be VERY happy if Google hid/ignored these kind of search-query sites. I have not once found any of these sites useful. Problem is the large gray area in which these sites operate and what I find useless, someone else might find useful.

StrawberryFrog 15y ago
> Actually there is no difference between search page result sites like the OP and what you said Uh, yes there is. http://www.somesite.com/index.html?q=foo http://www.somesite.com/index.html?q=foo is obviously a search (right down to the choice if q for query), but http://www.somesite.com/buy_foo_now.html http://www.somesite.com/buy_foo_now.html does not look like one. You might say that technically they could have the same back-end, but the web is flexible - just about any URL scheme could resolve to the same back-end. The difference is that the first is honest and the second one goes out of its way to hide what it does.
- ceejayoz 15y ago
  > Uh, yes there is. http://www.somesite.com/index.html?q=foo http://www.somesite.com/index.html?q=foo is obviously a search Obviously? Drupal's default non-rewritten URLs are index.php?q=foo, where foo is the page path. `q` can and does stand for more than one thing.
robryan 15y ago
Would be easy for the google bot to work this out to, replace the URL part that looks like a dynamic variable with different values and see if it returns a page for them all like a search engine would. Surprised it doesn't appear to do this.

joshu 15y ago

Search results are also not supposed to be indexed. I believe this is actually in the guidelines.

Matt_Cutts 15y ago
Yup. Search for [quality guidelines] and they're at http://www.google.com/support/webmasters/bin/answer.py?answer=35769 http://www.google.com/support/webmasters/bin/answer.py?answe... . The relevant part is "Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don't add much value for users coming from search engines."
- j_col 15y ago
  Matt, I tried that recently with one of my own sites, and after a few days I got a warning in my Google Webmaster Tools dashboard that the bot could not access my /search URL because it was blocked in my robots.txt, and I should take action to correct it. So I then unlocked my /search URL, which is in violation of the above guideline, but made the error in Google Webmaster Tools go away. These conflicting messages from Google are very confusing!
  - vanessafox 15y ago
    Hi j_col- In that case, Google Webmaster tools is not actually reporting an error. That's a report to show you what URLs Google tried to crawl but couldn't (due to being blocked) so you can review it and ensure that you are not accidentally blocking URLs that you want to have indexed. I agree that it's confusing in that the report is in the "crawl errors" section. (I built Google webmaster tools so this confusion is entirely my fault; but I don't work at Google anymore so sadly I can't fix this.)
    - j_col 15y ago
      Thanks for the response, I will block my /search URL once more via robots.txt and will ignore the warnings in the Webmaster Tools.