3 ms·
“…treat the absence of one as a lack of consent. ” - do you have a source for this? Their documentation states otherwise https://developers.google.com/search/d
by rezbull 5y ago
“…treat the absence of one as a lack of consent. ” - do you have a source for this?
Their documentation states otherwise https://developers.google.com/search/docs/advanced/robots/robots-faq#h01 https://developers.google.com/search/docs/advanced/robots/ro...
- otterley 5y agoYou are correct; my mistake. Nevertheless S3 returns a 403 (unauthorized) response for robots.txt by default which causes Google not to index it.