4 ms·
Googlebot snarfing URLs from the ether?
So the other day I had placed a private CGI script on my personal server for testing. (No links to the URL; directory redirects to my home page.) It crashed my server, so I rebooted it and decided to come back later.
Two days later, I get an e-mail from Google's Webmaster tools (which I have configured on that server) that my site is down. Indeed, it needs another reboot, as if someone accessed that private URL. But surely no bot could know that URL exists, right? Checking my server logs I find this (1.2.3.4 is my IP address):
1.2.3.4 - - [31/May/2013:01:01:41 +0000] "GET /~chris/avcc.cgi HTTP/1.1" 200 283 "-" "Opera/9.80 (X11; Linux x86_64; U; en) Presto/2.10.289 Version/12.02"
66.249.73.230 - - [31/May/2013:01:02:11 +0000] "GET /robots.txt HTTP/1.1" 200 477 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.73.230 - - [31/May/2013:01:02:11 +0000] "GET /~chris/avcc.cgi HTTP/1.1" 200 291 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
That is, mere seconds after I had first accessed a URL the world had never seen before, Googlebot decided to crawl my site (as indicated by the robots.txt hit), and then access the URL that was not linked anywhere.
My only explanation is that somewhere between my home computer and my (off-site) server, a Google-affiliated network appliance sniffed my initial GET and sent the URL to Googlebot to be indexed within 30 seconds.
Has anyone else experienced this?
(Yes I know security through obscurity, should use robots.txt, yadda yadda.)
- stray 13y agoDid you access that URL with Google Chrome mere seconds before googlebot miraculously found it?
- galacticvoid 13y agoOr does the browser your using have instance results from the address bar and the search engine is Google?
- colanderman 13y agoYes, though I've tried testing for this, but I only see requests from my browser (Opera) for incomplete URLs.
- gee_totes 13y agoWait, are you saying the Google Chrome will send URLs back to Google for indexing? I did not know this
- stray 13y agoI'm not saying it as a statement of fact. But rather as a plausible explanation for what OP experienced. But if I were google I would definitely give people free tools with which they could perform, with actual human intelligence, tasks I had found difficult.
- colanderman 13y agoNo, I'm using Opera (see the log). It's possible Opera is sending such info as well (it uses Google for autocompletion), but I've tried reproducing this and I don't see evidence of it. (I do see autocompletion requests, but never for complete URLs.)
- stray 13y agoThat is surprising and somewhat troubling.
- mattsah 13y agoDid you use Chrome to access the URL?
- colanderman 13y agoNo, I'm using Opera.