OP here. I'm not sure about the details in your link, but basically my understanding lines up with [1]; robots.txt isn't guaranteed to be respected, but general

OP here. I'm not sure about the details in your link, but basically my understanding lines up with [1]; robots.txt isn't guaranteed to be respected, but generally is. FWIW, what I specifically have in robots.txt is User-agent: * Disallow: / which seems to work well for me so far (i.e., I do not find my house documentation site on any search engine). [1]: https://developers.google.com/search/docs/crawling-indexing/robots/intro#understand-the-limitations-of-a-robots.txt-file https://developers.google.com/search/docs/crawling-indexing/...