5 ms·
HN can always change their robot.txt and prevent google from crawling. Many of copy left friends do that to their blog and site to prevent google/bing from inde
by factormeta 3y ago
HN can always change their robot.txt and prevent google from crawling. Many of copy left friends do that to their blog and site to prevent google/bing from indexing them.
- xp84 3y agoJust so you know, that doesn’t stop Google from indexing you anymore. It only stops Googlebot from crawling. In other words it’s strictly a place to deny permission or set limits on a technical level. If you do this they will still have what they already crawled and if there are links to it on the Internet they will keep it in the index. If you don’t want something to be in Google on a policy level, you now have to add the noindex meta tag on every page.
- sitkack 3y agoThis is really counter productive towards furthering the cause.