3 ms·
No, that's not how Google's scraper works. They only follow publicly-exposed links, and they check your robots.txt file to see if they should ignore any links t
by DougWebb 9y ago
No, that's not how Google's scraper works. They only follow publicly-exposed links, and they check your robots.txt file to see if they should ignore any links they might have come across that you intended to be private.
If you accidentally put a link on your public website to content you didn't want public, and Google indexes it, then someone else fetches it, yeah, that's your fault for exposing it. It's also your responsibility for telling Google to remove it from their index. But if someone is poking around to see if urls are valid (eg: the /.ssh/id_rsa requests my website gets) that's breaking and entering, even though I'd be negligent for leaving my keys on the webserver like that.
- alkonaut 9y agoI know - but let's assume that since I and google could reach the document it was inadvertently put on a public server. Since it was a mistake it won't be filtered in robots.txt either. For example, in the weeks before an Apple event, people tend to visit presumed unannounced product pages (by guessing) etc.
- crazyhatfish 9y agoActually, I don't think they state that is the only way they find content. I've had plenty of sites with obscure domain names for development and they always seem to find their way on to Google some how. I imagine in addition to public URLs they also use URLs entered into Chrome (which isn't that public) and since they host DNS they probably crawl domain names requested through them too. Legally I'm not sure what they stand on to do that but they're big enough now so I guess they don't worry too much about that nowadays.