6 ms·
Ask HN: How to stop Google indexing dynamic search pages?
Hey HN folks,
A few months ago I received a manual action penalty from Google as they detected spam pages on our domain. The problem was that when people were searching on our site they are directed to a page with the following:
https://$domain/search?query=$QUERY
Some users (most likely bots) are generating huge spam searches on our search page and somehow Google is indexing these and there are no inbound links to these pages (at least I cannot find any).
To resolve this I did the following:
* On our search page I set the following header: X-Robots-Tag: noindex (based off of the documentation here https://developers.google.com/search/reference/robots_meta_tag).
* Submitted URLs to be dropped from Google Index via Webmaster console
* Submitted 3 reconsideration requests to Google to avoid the penalties
In theory this should stop all search pages being indexed (as they all contain the noindex header) and it has helped drop the number of indexed pages marked as spam by 99% however we still have a significant number of urls marked as spam and so our site has a penalty from Google.
Has anyone had this issue before? How can I stop these pages becoming indexed when I have the noindex header set _and_ if you search the spam urls there are no inbound links to them?
Any help appreciated folks!
- itamarst 8y agoMaybe also add a robots.txt? http://www.robotstxt.org/ http://www.robotstxt.org/
- scalesolved 8y agoWould adding robots.txt help? I cannot blanket ban the page as it needs to display to the user with 0 or more results and my understanding is that whilst robots.txt would take precedence for index settings as I have it defined in the header for that page it ought to achieve the same.
- wejick 8y agoI think adding rule for /search* to your robot is easier way to block google bot
- dougunplugged 8y agoDon't do this until Google sees your new robots meta directives (or canonical tag) on these pages, that way they will drop from the index. Then you can add this to your robots.txt to prevent it being crawled again.
- dgranda 8y agoBased on my experience: A.- I would also add "nofollow, noarchive" tags [1] to your X-Robots-Tag header: - "nofollow" -> do not to follow (i.e., crawl) any outgoing links on the page. - "noarchive" -> prevents Google from showing the Cached link for a page. B.- I would specify in Search Console (former Webmaster Console) how should Google handle "query" parameter [2] C.- Prevent those spam searches by blocking source IP address, User-Agents, combinations of both, etc. Good luck! [1] https://support.google.com/webmasters/answer/79812?hl=en https://support.google.com/webmasters/answer/79812?hl=en [2] https://www.google.com/webmasters/tools/crawl-url-parameters?hl=en&siteUrl=https://<domain>/ https://www.google.com/webmasters/tools/crawl-url-parameters...
- scalesolved 8y agoThanks for the suggestions! A) I'm going to add the nofollow and noarchive to see if that helps the issue. B) I've already set the search console to ignore the query parameter but I'm still getting new spam results coming in. C) I've been looking into this but so far the meta information for the spam requests is not consistent and it's tricky to identify so far. Thanks for the help and the luck, I think I'll need it!
- detaro 8y agoAre they still being added newly, or have just not been purged from Google index yet?
- scalesolved 8y agoThere are still new spam search queries being indexed.
- detaro 8y agoDo you have a robots.txt entry that's stopping Google from fetching them? That can counter-intuitively cause Google to index pages.
- scalesolved 8y agoI do not have a disallow set for this page but based off of this page it seems perhaps I might have to set one: https://www.deepcrawl.com/blog/best-practice/noindex-disallow-nofollow/ https://www.deepcrawl.com/blog/best-practice/noindex-disallo... Specifically the part: >Noindex (robots.txt) + Disallow: This prevents pages appearing in the index, and also prevents the pages being crawled. However, remember that no PageRank can pass through this page.
- detaro 8y agoGoogle specifically warns that having a Noindex header doesn't work if there is robots.txt disallow (since the crawler never sees the noindex, since it obeys the disallow), that's why I asked.
- r1ch 8y agoThere's a non standard Noindex: /URL that Google accepts in robots.txt. Unfortunately it breaks many other crawlers which don't understand it, so you have to rely on user agent sniffing.
- stevenicr 8y ago
- sebst 8y agoYou could also annotate your page. https://schema.org/SearchResultsPage https://schema.org/SearchResultsPage Edit: Maybe it is also worth annotating the search field (https://developers.google.com/search/docs/data-types/sitelinks-searchbox https://developers.google.com/search/docs/data-types/sitelin...) so that google can match it against your search results page.
- helij 8y agoYou need to add <meta name="robots" content="noindex, follow"> to the <head> section of all your search results pages. You want robots NOT to index pages but to still follow links on your search pages. Create clean sitemap.xml file and submit it to Search Console. Another way is to just canonicalize all search results pages to your search page. With Google and these things time is involved. Once it's in the index it will take time to properly clean everything up. How was the traffic before this happened? Did the website rank for any decent keyword? Sometimes when this happens the smart thing to do is to just start from scratch with a new domain. If you want more extensive help email me.
- tangue 8y agoYou should use the canonical tag. Moz has a good page on how it works. https://moz.com/blog/canonical-url-tag-the-most-important-advancement-in-seo-practices-since-sitemaps https://moz.com/blog/canonical-url-tag-the-most-important-ad...
- eddflrs 8y agoAdding <meta name="robots" content="noindex" /> to each page should work. Also as a heads up, having an entry in robots.txt to disallow is not enough since pages can still be indexed if they can be navigated from anywhere else on the web.
- aidos 8y agoI thought robots.txt was meant to be pulled from the domain and honoured anyway. At least that’s what used to happen. Just because someone links to you doesn’t mean the spiders should crawl all the content
- computator 8y agoCan anyone answer a related question: Are you penalized for not running Google Analytics and/or Google Webmaster tools? In other words, if you have a clean website with no analytics whatsoever, is your ranking likely to be worse?
- GrungeMarx 8y agoHaving worked previously in the SEO-industry, I would say that you are not penalized for not using the Google Webmaster toolbox (analytics, WMT etc.) - however, by not having an GWT account Google has no way of getting in contact with you if they think you're doing some shady/spammy or if they penalized you. That's the main reason to have a GWT account at all, really.
- Topgamer7 8y agoAnyone who doesn't work at Google wouldn't be able to with absolute certainty be able to answer this question. Anyone who does work at Google wouldn't be allowed to answer this question. However page rank is widely attributed to mainly based on back links.
- computator 8y agoThanks for the answer. But couldn't you test it by setting up two very similar websites (not identical because you'd presumably get penalized for duplicating content), in which one uses Google Analytics and the other doesn't, and then watch the ranking over time. If the analytics site does significantly better than the other, that would strongly indicate that Google gives it preference.
- Topgamer7 8y agoIt would be a decent undertaking to build two websites with comparable but different content to test search rank. It is also very probably that the search ranking algorithm is nondeterministic.
- Jedi72 8y agoIf not using GA doesnt affect your page rank, I don't see why Google would need (or even want) to be secretive about it. Only if it does hurt your ranking (how is that huge antitrust lawsuit today feeling Big G??) will they need to be all secretive about it.
- Jaruzel 8y agoRegister for Google Webmaster tools. There's an option in there to exclude links that have dynamic parameters. You can define the parameters you want it to ignore.
- jacquesm 8y agoHilarious how Google thinks they are now in editorial control of your content to the point where you are on the hook for fixing their bugs. You're being treated as a wayward content provider, rather than that they should be happy to get the benefit of your content to index.
- Fnoord 8y agoProblem, as usual, lies with default being opt-out instead of opt-in. They did the same with WAPs. Now mine need to append _nomap.
- kiallmacinnes 8y agoWhile I agree, opt-in would certainly be better.. I really see no way to transition from the current state, to 100% opt in. And, I think opt-out was the right choice at the beginning - nobody would have updated their sites to suit the startup search engine. That said, Google & co could set a date, whereby new content / pages will not be indexed unless its marked as indexable. This would allow historic content to remain indexed, and new content be opt-in at the expense of any new search engines being unable to index historic content. How can they tell if the content has not opted in, or is just no longer actively maintained without having a pre-existing index?
- known 8y agoYou can restrict in .htaccess
- officialchicken 8y agoThat might work, but only if you use apache. If there is a CDN or a different server or proxy (nginx, varnish) it's probably better to rely on meta tags, robots.txt, and/or canonical tag.
- emilfihlman 8y agoHeh, I ran into a similar issue previously: https://news.ycombinator.com/item?id=16302821 https://news.ycombinator.com/item?id=16302821 GoogleBot is broken.
- lgats 8y agoBlocking the search function in robots.txt may help as well. User-agent: * Disallow: /search Disallow: /search