5 ms·
Author should update their robots.txt for Googlebot as well. It is not clear if noindex means "notrain" too. The entire webpage has to be read in and parsed for
by chomp 3y ago
Author should update their robots.txt for Googlebot as well. It is not clear if noindex means "notrain" too. The entire webpage has to be read in and parsed for Google to extract that meta tag. However, robots.txt should stop the crawler before it proceeds to the rest of your site.
- DanHulton 3y agoThe author mentioned she was going to block Googlebot, only not yet, in order to make sure it can crawl the site again to get the 'noindex' instructions.