3 ms·

What I'm hearing is that if I tweak robots.txt I can exclude my site from google? Excellent news!

• aendruk 9mo ago
  Was that not possible before?
• deleted 9mo ago
  [deleted]
• sofixa 9mo ago
  This is literally the point of robots.txt. It was created to allow site owners to configure how and which parts of their website can be scraped by what bot, and all the "decent" ones (Google, Bing) respect it.
  • bflesch 9mo ago
    Spoiler: They don't.
    • edwinjm 9mo ago
      Google, Bing do
• dazc 9mo ago
  no-index is a thing.
  • tremon 9mo ago
    no-index is per individual page, not for an entire domain, IIUC?
    • snowwrestler 9mo ago
      You can pretty easily send no-index for an entire site by configuring it as a site-wide HTTP header.