5 ms·
Why not use robots.txt instead of littering your html with googlebot instructions?
by bouke 5y ago
Why not use robots.txt instead of littering your html with googlebot instructions?
- TheChaplain 5y agoYes, pretty sure this is the way to go. You can even tell which bots are allowed to index and not.
- vadfa 5y agoOr even better, iptables rules :P
- TedDoesntTalk 5y agoDoesn’t that mean you have to know every IP Address used by Google bot now and in the future?
- judge2020 5y agoThe way to check googlebot (in a way that will be resistant to expansion of Googlebot's IP ranges in the future) is to perform hostname lookup, with dns lookup as well to verify that the rDNS isn't a lie: https://developers.google.com/search/docs/advanced/crawling/verifying-googlebot https://developers.google.com/search/docs/advanced/crawling/...
- lucb1e 5y agoIndeed, this was one of the things I considered (note I'm not OP), but then I didn't really want to rely on DNS. https://duckduckgo.com/?q=it's+always+DNS https://duckduckgo.com/?q=it's+always+DNS
- AshamedCaptain 5y agoNot a very hard problem; after all, many websites allow full access to Googlebot IP ranges yet show a paywall to everyone else (including competing search engines). I also happen to ban Google ranges on multiple less-public sites specially since they completely ignore robots.txt and crawl-delay.
- chihuahua 5y agoIs that how archive.vn works? I've always wondered how they are able to get the full text of paywalled sites like Wall Street Journal who give 0 free articles per month.
- tao_oat 5y agoHi, author here. Google stopped supporting robots.txt [edit: as a way to fully remove your site] a few years ago, so these meta tags are now the recommended way of keeping their crawler at bay: https://developers.google.com/search/blog/2019/07/a-note-on-unsupported-rules-in-robotstxt https://developers.google.com/search/blog/2019/07/a-note-on-...
- new_guy 5y agoDid you actually read your link? That's not at all what it says.
- tao_oat 5y agoTo be clear, stopped supporting robots.txt noindex a few years ago. Combined with the fact that Google might list your site [based only on third-party links][1], robots.txt isn't an effective way to remove your site from Google's results. Sorry, could have been clearer. [1]: https://developers.google.com/search/docs/advanced/robots/intro https://developers.google.com/search/docs/advanced/robots/in...
- dd82 5y ago>noindex in robots meta tags: Supported both in the HTTP response headers and in HTML, the noindex directive is the most effective way to remove URLs from the index when crawling is allowed. Seems clear enough to me
- jen20 5y agoQuote from the linked article: “ For those of you who relied on the noindex indexing directive in the robots.txt file, which controls crawling, there are a number of alternative options:” The first option is the meta tag. It does mention an alternative directive for robots.txt, however.
- ghassanmas 5y agoWhat about the blocking google bot by their IPs, also combined with user-agent wouldn't that stop the crawlers Google crawlers IPs https://www.lifewire.com/what-is-the-ip-address-of-google-818153 https://www.lifewire.com/what-is-the-ip-address-of-google-81...
- walshemj 5y agorobots.txt stops crawling - you can get indexed via other mechanisms. You want no index robots tags on all your pages and let google see those. You can use GSC (Google search console) to remove a site / page from the index
- intricatedetail 5y agoI have disallowed all robots in my robots.txt and still project shows up in the search.
- bhartzer 5y agoGoogle will still index the url even if you block them from crawling the page via robots.txt. They will index the lane, and it can still rank well. Google just puts up a message in the results saying they’re not allowed to crawl the page.