3 ms·

What I'm hearing is that if I tweak robots.txt I can exclude my site from google? Excellent news!

• aendruk 10mo ago
  Was that not possible before?
• deleted 10mo ago
  [deleted]
• sofixa 10mo ago
  This is literally the point of robots.txt. It was created to allow site owners to configure how and which parts of their website can be scraped by what bot, and all the "decent" ones (Google, Bing) respect it.
  • bflesch 10mo ago
    Spoiler: They don't.
    • edwinjm 10mo ago
      Google, Bing do
• dazc 10mo ago
  no-index is a thing.
  • tremon 10mo ago
    no-index is per individual page, not for an entire domain, IIUC?
    • snowwrestler 10mo ago
      You can pretty easily send no-index for an entire site by configuring it as a site-wide HTTP header.