3 ms·
Quote from the linked article: “ For those of you who relied on the noindex indexing directive in the robots.txt file, which controls crawling, there are a num
by jen20 5y ago
Quote from the linked article:
“ For those of you who relied on the noindex indexing directive in the robots.txt file, which controls crawling, there are a number of alternative options:”
The first option is the meta tag. It does mention an alternative directive for robots.txt, however.
- ghassanmas 5y agoWhat about the blocking google bot by their IPs, also combined with user-agent wouldn't that stop the crawlers Google crawlers IPs https://www.lifewire.com/what-is-the-ip-address-of-google-818153 https://www.lifewire.com/what-is-the-ip-address-of-google-81...
- benatkin 5y agoThat will stop the crawlers but you could still show up in the search results, because of other web pages. From GP: > If other pages point to your page with descriptive text, Google could still index the URL without visiting the page
- intricatedetail 5y agoMeta tag implies robot is going to fetch the page, effectively at very least using up you bandwidth. It should be illegal.