4 ms·

Surprisingly this person didn't even bother adding a noindex/robots.txt measure to the website. So "deleted forever" is only as good as someone not saving the

• saimiam 6y ago
  After reading your comment, I realized that I had to do this on my own site as well. Harsh but valuable comment (to me).
  • rriepe 6y ago
    Mine got nuked somewhere along the way too.
• skinkestek 6y ago
  Really useful comment that I had to think twice not to downvote because of its snarkfulness.
• adamhearn 6y ago
  Can’t you just ignore robots.txt?
  • saimiam 6y ago
    Well behaved crawlers don't. Products like Google Search and Bing have a self-interest in supporting robots.txt because the alternative is annoyed webmasters suing them for not providing an opt-out due to which their unsecured credentials are a quick search away. In fact, I believe the robots.txt protocol was devised in consultation with search engines.
    • mrkramer 6y ago
      A lot of webmasters are amateurs who expose their credentials even when robot rules are respected. Google dorking is a way to find that credentials and other sensitive data and information.
    • mxxx 6y ago
      Unless I’m mistaken, google don’t actually respect robots.txt any more. They recommend the use of some meta tag instead, from memory.
      • yarcob 6y ago
        Their docs say they do: https://developers.google.com/search/reference/robots_txt https://developers.google.com/search/reference/robots_txt They do say that they ignore robots.txt for user-initiated actions eg. Google Translate, which makes sense in my opinion.
        • mxxx 6y ago
          Ah yes you’re right, my apologies. They don’t support the noindex directive inside a robots.txt any more.
• mxxx 6y ago
  Archivers aren’t going to crawl it unless someone actually provides the URL. At which point it’s the same as screenshooting and putting somewhere public, no?
• n_david 6y ago
  Hey, thanks for letting me know this. This was definitely an oversight by me and I'll add this in.