5 ms·
An alternative would be to check for the browser user agent and delete the website right at that point and return a 404 page to the Google crawler bot. Then Goo
by frik 12y ago
An alternative would be to check for the browser user agent and delete the website right at that point and return a 404 page to the Google crawler bot. Then Google won't have a static copy of the website.
- ForHackernews 12y agoThey do tell google not to save a static copy: > the NOARCHIVE meta tag is specified which prevents the Googles from caching their own copy of the content.
- ikeboy 12y agoWouldn't that only prevent the user from seeing the cache? I mean, if it's indexed, then google must have it cached, right?
- adanto6840 12y agoThat meta tag prevents Google from publicly showing their cached version of the page. In practice this means the "Cached" link, within the results, doesn't appear when a given page asks Google to NOARCHIVE -- which I believe can be 'asked for' via either the meta tag or via a special response header. Edit: Yeah, 'noarchive' can be specified via the meta tag or via header. Also available to you are a handful of other directives such as NoIndex, NoFollow, NoArchive, NoSnippet, NoTranslate, etc... See these links for more in-depth info about the directives & which search engines support what: Directives & Usage in Meta tags - http://noarchive.net/meta/ http://noarchive.net/meta/ Usage in Response Headers - http://noarchive.net/xrobots/ http://noarchive.net/xrobots/
- LukeB_UK 12y agoThe problem with that is that you could spoof the user agent.
- jonlucc 12y agoBut Google probably doesn't bother, right?
- LukeB_UK 12y agoWhat I meant is that a human could spoof the user agent and pretend to be Googlebot.
- desdiv 12y agoGooglebot's identity can the authenticated to prevent spoofing: https://support.google.com/webmasters/answer/80553?hl=en https://support.google.com/webmasters/answer/80553?hl=en
- haneefmubarak 12y agoActually, you could do a reverse IP lookup against any user agent claiming to be googlebot followed by a forward IP lookup against the domain name you were returned. Legitimate googlebots will be in the *.googlebot.com space. Source: https://support.google.com/webmasters/answer/80553?hl=en https://support.google.com/webmasters/answer/80553?hl=en
- desdiv 12y agoYour approach is "a website that irrevocably deletes itself once indexed by Google". What OP has done is "a website that irrevocably deletes itself once Google decided to publicly reveal the fact that it indexed said website". OP's approach has no way of knowing when the site was indexed. It's conceivable that Google indexed it on the very first day and decided not to share it publicly until 21 days later.
- BorisMelnik 12y agohah, good point for all intents and purposes (for most) indexed is when it appears in the search results
- lukeschlather 12y agoEven if the request that claims to be from Googlebot is actually the Googlebot (which it might not be,) that doesn't guarantee the site is indexed. It's impossible to know when the site is indexed without direct access to Google's index.
- nostrademons 12y agoTechnically, the former is when it is "crawled" and the latter is when it is "indexed". In practice, since 2010 these two events have generally been separated by minutes.
- tyrust 12y agoIf you really want to get "technical", then the first one is when the site is "crawled" and the latter is when it's "served". "Indexing" happens in-between the two.