4 ms·

Even if it's behind a paywall, there is benefit in crawling it. For instance, if you happened to be a logged WSJ user, then google could show you the result ba

• dragonwriter 9y ago
  Googlebot does not crawl the web separately for each user with a copy of that user's credentials borrowed from their browser. Aside from the privacy and security issues, it would require Google to multiply it's search resources by the number of users.
  • 2_listerine_pls 9y ago
    > Googlebot does not crawl the web separately for each user with a copy of that user's credentials borrowed from their browser. I did not say that. I said google could display the results to which you have access, based on your cookies. Assuming they had access to crawl the whole article.
    • CydeWeys 9y ago
      That would require Google to be in the business of knowing every site that you have a membership in. It is not possible for Google to get access to cookies for other sites, anyway. This is a pretty fundamentally important security restriction that browsers implement to protect you from nefarious sites. So it isn't possible for Google to know which sites you have paid accounts with unless you explicitly tell it.
• Flenser 9y ago
  > Even if it's behind a paywall, there is benefit in crawling it. How would that work? Would Google create or be given a login for WSJ? That would benefit WSJ as an incumbent news provider at the expense of startups too new or small to get special treatment from Google. How would a new website indicate to all search crawlers (so google doesn't befit at the expense of other search engines) how to get access to it's pay-for content in such a way that end-users can not also pretend to be seach crawlers and get access to the same content?