5 ms·

if you can find those cams via a google search, doesn't that mean they are linked from some other public site (which has also been indexed) which would indicate

if you can find those cams via a google search, doesn't that mean they are linked from some other public site (which has also been indexed) which would indicate they were left public intentionally (at least most of them)? If I would set up a web-accessible cam without password protection, how would google find it? Its a crawler right? It doesnt just searches for random IPs and tries to connect to them. I always was under the assumption that there is a pretty big part of the internet which is just not indexed by the major search engines (thus more or less private).

0x0 14y ago
Google is amazingly good at digging up sites out of nowhere. I wonder if it is a combination of URLs passing through Chrome, GMail, any android phone, and so on. It's always a hassle keeping staging/dev sites out of the index if you're not careful with all the right meta noindex and robots.txt tags. (robots.txt with disallow all, on its own, won't keep sites/URLs from showing up in the results, at best just hide the cached body text summary below the link)
- diminoten 14y ago
  > I wonder if it is a combination of URLs passing through Chrome, GMail, any android phone, and so on. That would be incredibly alarming, and quite possibly the largest breach of trust perpetrated by a company so far this decade.
  - hdevalence 14y ago
    When you type into the chrome omnibox, it sends your keystrokes to Google to give you search suggestions. Just using those would be sufficient, and I don't think they really hide that it's sending your input along. If someone does a Google search for a URL, we expect it to get added to the index; why is it different when the search occurs in the omnibox rather than their web interface?
    - diminoten 14y ago
      URLs don't go through google when you type them in Chrome's URL box, they go directly to the address. Chrome only sends things to Google that it can't interpret as a URL.
      - randallu 14y ago
        Actually it looks like URLs do go to google's suggestion service as you type into Chrome's URL box. I tried typing "http://then http://then and it suggested a UPS package URL and "thenicestplaceontheinter.net". I've not been to either of those pages before (I use Chrome for testing, so I'm not signed into it, etc).
        diminoten 14y ago
        I can tell you assertively that, if you capture packets while using Google Chrome's address bar for URLs, it does not send the data to Google.
  - MiguelHudnandez 14y ago
    While having the Bing toolbar installed in IE, any URL one types or visits is submitted to Microsoft, and they actively use this data to tune their Bing search results. http://www.wired.com/business/2011/02/bing-copies-google/ http://www.wired.com/business/2011/02/bing-copies-google/ I agree it'd be alarming and terrible, but hardly a new development. Edit: it's doubtful that an e-mail provider would automatically fetch links from e-mails -- think about them clicking 'unsubscribe' links and links to reject the transfer of domain names. It would break in very obvious ways. IMs and texts, on the other hand, might be more opaque to that kind of meddling.
    - 0x0 14y ago
      It'd be interesting to set up a wildcard dns *.some-experiment.example.com, and send various http://links-via-gmail.some-experiment.example.com/somepath http://links-via-gmail.some-experiment.example.com/somepath , http://links-via-skype.some-experiment.example.com/anotherpath http://links-via-skype.some-experiment.example.com/anotherpa... through a bunch of services, and see which domain names and which full URLs show up in the logs!
      - darklajid 14y ago
        Go for it! That sounds like a great idea. Make sure that your results can be tracked and provide as much information as possible and you got a nice project here..
        0x0 14y ago
        Maybe I will, but if anyone else feels like putting in the effort, go ahead, too :)
        MiguelHudnandez 14y ago
        I just registered hnypot [dot] info for a few bucks and will see if I can get wildcard DNS running with some tracking. Haha, I don't want to type the name as a link until I get the tracking going... If anybody wants to collaborate or you just want an NS delegation off that name to try to roll your own, just let me know!
  - 0x0 14y ago
    Good point, come to think of it, the last time I had to deal with this, it could have been caused by a number of entries into google, such as embedding google analytics .js even on the staging site, or running a test on the google page speed tools, etc. But they certainly have a HUGE amount of opportunity to snap up new URLs across all their services.
  - deleted 14y ago
    [deleted]
to3m 14y ago
People sometimes leave their site logs public, which I suspect means google could find links to private pages in the referer logs. (Of course, people should be relying on more than obscurity to hide their private pages! - but I believe people have been known to make mistakes and/or not know what they're doing.)
pavel_lishin 14y ago
Some IRC networks or channels make their logs publicly accessible - so if you're showing off your parakeet on camera one day, that URL might be logged.

0x0 14y ago

Google is amazingly good at digging up sites out of nowhere. I wonder if it is a combination of URLs passing through Chrome, GMail, any android phone, and so on. It's always a hassle keeping staging/dev sites out of the index if you're not careful with all the right meta noindex and robots.txt tags. (robots.txt with disallow all, on its own, won't keep sites/URLs from showing up in the results, at best just hide the cached body text summary below the link)

diminoten 14y ago
> I wonder if it is a combination of URLs passing through Chrome, GMail, any android phone, and so on. That would be incredibly alarming, and quite possibly the largest breach of trust perpetrated by a company so far this decade.
- hdevalence 14y ago
  When you type into the chrome omnibox, it sends your keystrokes to Google to give you search suggestions. Just using those would be sufficient, and I don't think they really hide that it's sending your input along. If someone does a Google search for a URL, we expect it to get added to the index; why is it different when the search occurs in the omnibox rather than their web interface?
  - diminoten 14y ago
    URLs don't go through google when you type them in Chrome's URL box, they go directly to the address. Chrome only sends things to Google that it can't interpret as a URL.
    - randallu 14y ago
      Actually it looks like URLs do go to google's suggestion service as you type into Chrome's URL box. I tried typing "http://then http://then and it suggested a UPS package URL and "thenicestplaceontheinter.net". I've not been to either of those pages before (I use Chrome for testing, so I'm not signed into it, etc).
      - diminoten 14y ago
        I can tell you assertively that, if you capture packets while using Google Chrome's address bar for URLs, it does not send the data to Google.
- MiguelHudnandez 14y ago
  While having the Bing toolbar installed in IE, any URL one types or visits is submitted to Microsoft, and they actively use this data to tune their Bing search results. http://www.wired.com/business/2011/02/bing-copies-google/ http://www.wired.com/business/2011/02/bing-copies-google/ I agree it'd be alarming and terrible, but hardly a new development. Edit: it's doubtful that an e-mail provider would automatically fetch links from e-mails -- think about them clicking 'unsubscribe' links and links to reject the transfer of domain names. It would break in very obvious ways. IMs and texts, on the other hand, might be more opaque to that kind of meddling.
  - 0x0 14y ago
    It'd be interesting to set up a wildcard dns *.some-experiment.example.com, and send various http://links-via-gmail.some-experiment.example.com/somepath http://links-via-gmail.some-experiment.example.com/somepath , http://links-via-skype.some-experiment.example.com/anotherpath http://links-via-skype.some-experiment.example.com/anotherpa... through a bunch of services, and see which domain names and which full URLs show up in the logs!
    - darklajid 14y ago
      Go for it! That sounds like a great idea. Make sure that your results can be tracked and provide as much information as possible and you got a nice project here..
      - 0x0 14y ago
        Maybe I will, but if anyone else feels like putting in the effort, go ahead, too :)
        MiguelHudnandez 14y ago
        I just registered hnypot [dot] info for a few bucks and will see if I can get wildcard DNS running with some tracking. Haha, I don't want to type the name as a link until I get the tracking going... If anybody wants to collaborate or you just want an NS delegation off that name to try to roll your own, just let me know!
- 0x0 14y ago
  Good point, come to think of it, the last time I had to deal with this, it could have been caused by a number of entries into google, such as embedding google analytics .js even on the staging site, or running a test on the google page speed tools, etc. But they certainly have a HUGE amount of opportunity to snap up new URLs across all their services.
- deleted 14y ago
  [deleted]

to3m 14y ago

People sometimes leave their site logs public, which I suspect means google could find links to private pages in the referer logs. (Of course, people should be relying on more than obscurity to hide their private pages! - but I believe people have been known to make mistakes and/or not know what they're doing.)

pavel_lishin 14y ago

Some IRC networks or channels make their logs publicly accessible - so if you're showing off your parakeet on camera one day, that URL might be logged.