4 ms·

So, how did Google become aware of the existence of these URLs in the first place? I seriously doubt they're linked from another Facebook page. Is Google har

So, how did Google become aware of the existence of these URLs in the first place? I seriously doubt they're linked from another Facebook page. Is Google harvesting links from secure pages using their toolbar or something? Are people's personal mails leaking through other means? I noticed that a few of them are indexed by Google because someone decided to reprint the email - URLs and all - to their blog. But that's a rare exception,and obviously not the case with the author of the article.

tcoxon 16y ago
It's simple: Google has access to the referrer URL when you go onto their website. It's well known that they index these URLs. If you go from that Facebook page straight to Google, it'll get indexed.
- deleted 16y ago
  [deleted]
- deleted 16y ago
  [deleted]
- sounddust 16y ago
  In what situation would the HTTP referrer header report Google when clicking away from a Facebook link? Not only is there no Google search field on Facebook, but Facebook also reroutes links through an intermediary page in order to hide the actual referring page.
  - tcoxon 16y ago
    I had assumed that Referer URLs were sent even if it wasn't from a link that was clicked. That's not true. (And yes, "Referer" is misspelt in the standard...) http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html > The Referer field MUST NOT be sent if the Request-URI was obtained from a source that does not have its own URI, such as input from the user keyboard.
noelchurchill 16y ago
Google has multiple ways of indexing, and following links is only one of them. I suspect Google found these pages by scanning the directory structure of facebook, indexing all pages in these subdirectorys, and finding more subdirectories to drill in to.
- eli 16y ago
  I think (though I'm too lazy to confirm) that the Google Toolbar browser plugin also reports URLs back to Google
  - sounddust 16y ago
    If Google is using their toolbar and/or Chrome to report private URLs back to Google for indexing, then that's a much more serious privacy violation than what's being reported here.
    - eli 16y ago
      A "private" URL should surely require a login or at least have a robots.txt, no?
      - masomenos 16y ago
        Plenty of sites use URLs like http://example.com/sd8n9g9g http://example.com/sd8n9g9g rather than require logins for private / semi-private material. What's wrong with that?
        nl 16y ago
        It's a pretty bad practice. I guess for semi-private material it's okay, but otherwise that's liable to leak. The biggest hole in it is referrers leaking to other sites, but I can think of other methods (eg probing the CSS history exposure bug).
      - michaelfairley 16y ago
petewarden 16y ago
If you look at http://facebook.com/robots.txt http://facebook.com/robots.txt there's actually a sitemap. It's protected, and may well not contain these pages, but I have heard that Google and other privileged partners have access to it.

tcoxon 16y ago

It's simple: Google has access to the referrer URL when you go onto their website. It's well known that they index these URLs. If you go from that Facebook page straight to Google, it'll get indexed.

deleted 16y ago
[deleted]
deleted 16y ago
[deleted]
sounddust 16y ago
In what situation would the HTTP referrer header report Google when clicking away from a Facebook link? Not only is there no Google search field on Facebook, but Facebook also reroutes links through an intermediary page in order to hide the actual referring page.
- tcoxon 16y ago
  I had assumed that Referer URLs were sent even if it wasn't from a link that was clicked. That's not true. (And yes, "Referer" is misspelt in the standard...) http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html > The Referer field MUST NOT be sent if the Request-URI was obtained from a source that does not have its own URI, such as input from the user keyboard.

noelchurchill 16y ago

Google has multiple ways of indexing, and following links is only one of them. I suspect Google found these pages by scanning the directory structure of facebook, indexing all pages in these subdirectorys, and finding more subdirectories to drill in to.

eli 16y ago
I think (though I'm too lazy to confirm) that the Google Toolbar browser plugin also reports URLs back to Google
- sounddust 16y ago
  If Google is using their toolbar and/or Chrome to report private URLs back to Google for indexing, then that's a much more serious privacy violation than what's being reported here.
  - eli 16y ago
    A "private" URL should surely require a login or at least have a robots.txt, no?
    - masomenos 16y ago
      Plenty of sites use URLs like http://example.com/sd8n9g9g http://example.com/sd8n9g9g rather than require logins for private / semi-private material. What's wrong with that?
      - nl 16y ago
        It's a pretty bad practice. I guess for semi-private material it's okay, but otherwise that's liable to leak. The biggest hole in it is referrers leaking to other sites, but I can think of other methods (eg probing the CSS history exposure bug).
    - michaelfairley 16y ago

petewarden 16y ago

If you look at http://facebook.com/robots.txt http://facebook.com/robots.txt there's actually a sitemap. It's protected, and may well not contain these pages, but I have heard that Google and other privileged partners have access to it.