6 ms·

Of course they let in google's crawler.

CydeWeys 9y ago
The whole purpose of making a webcrawler for a search engine is that you are crawling the content that a user will see upon clicking a link in search results.
- 2_listerine_pls 9y ago
  Even if it's behind a paywall, there is benefit in crawling it. For instance, if you happened to be a logged WSJ user, then google could show you the result based on your cookies.
  - dragonwriter 9y ago
    Googlebot does not crawl the web separately for each user with a copy of that user's credentials borrowed from their browser. Aside from the privacy and security issues, it would require Google to multiply it's search resources by the number of users.
    - 2_listerine_pls 9y ago
      > Googlebot does not crawl the web separately for each user with a copy of that user's credentials borrowed from their browser. I did not say that. I said google could display the results to which you have access, based on your cookies. Assuming they had access to crawl the whole article.
      - CydeWeys 9y ago
        That would require Google to be in the business of knowing every site that you have a membership in. It is not possible for Google to get access to cookies for other sites, anyway. This is a pretty fundamentally important security restriction that browsers implement to protect you from nefarious sites. So it isn't possible for Google to know which sites you have paid accounts with unless you explicitly tell it.
  - Flenser 9y ago
    > Even if it's behind a paywall, there is benefit in crawling it. How would that work? Would Google create or be given a login for WSJ? That would benefit WSJ as an incumbent news provider at the expense of startups too new or small to get special treatment from Google. How would a new website indicate to all search crawlers (so google doesn't befit at the expense of other search engines) how to get access to it's pay-for content in such a way that end-users can not also pretend to be seach crawlers and get access to the same content?
DougWebb 9y ago
Possible, but seems unlikely. To set that up the WSJ website would have to allow Googlebot access while denying others. Any filtering based on the url or HTTP headers would be discovered and abused by others. An approach based on a security token or IP filter could work, but would be un-managable on Google's side because of the scale of their spidering operation. It would be much more effective for them to use their position to force the WSJ to be an open website, or to accept that their paywalled content does not get indexed.
- devicenull 9y ago
  Just allow access from all of Google's IP space, as long as the user-agent contains "googlebot'. It's pretty trivial to do... Sure, some people could set up GCP instances and proxy it, but that's a very tiny percentage of people.
  - DougWebb 9y ago
    It might start out as a tiny percentage, but all it takes is one person setting it up and letting the world know about it. Then pretty soon the WSJ is faced with millions of people getting free content again. They're complaining about people accessing their articles via a Google search and then clearing cookies to reset their counters. That's hardly mainstream; browsers have been burying the clear-cookies functionality deeper and deeper over the years because it's seen as an advanced-user-only kind of thing. And yet the WSJ has millions of people doing it, enough to make an impact on their bottom line.
    - devicenull 9y ago
      I'd bet it's browing in incognito mode versus actually clearing cookies.
  - obstinate 9y ago
    Googlebot's IP range and GCP's IP ranges are disjoint.
  - detaro 9y ago
hackcasual 9y ago
That's a no-no. https://support.google.com/webmasters/answer/66355?hl=en https://support.google.com/webmasters/answer/66355?hl=en
- 2_listerine_pls 9y ago
  I went through the whole thing and it seems, from the video, that they could avoid cloaking by redirecting non-paid users to the sing-up page. This way WSJ would be showing the same (www.wsj.com/Bezos-with-hair) to every user. *based on the video
  - jacquesm 9y ago
    > sing-up page. That's a new one. Buy a subscription to the WSJ now for a song.
  - squeaky-clean 9y ago
    In the video, he stresses pretty hard that all forms of cloaking are disallowed, even if not malicious or deceiving. Unless the Googlebot has a paid subscription to WSJ that's still cloaking, as you're showing Googlebot a different page than a regular user. Google's rules and help documents are spread all over, but here's some from Google News about subscriptions: "If you prefer this option, please display a snippet of your article that is at least 80 words long and includes either an excerpt or a summary of the specific article. Since we do not permit "cloaking" -- the practice of showing Googlebot a full version of your article while showing users the subscription or registration version -- we will only crawl and display your content based on the article snippets you provide." edit, forgot link: https://support.google.com/news/publisher/answer/40543?hl=en https://support.google.com/news/publisher/answer/40543?hl=en
- nso 9y ago
  I guess what this one infact states is that what they were doing was okay. In the video the guy explains that while you are not allowed to treat gbots request in a way no other people are treated, it is okay to differentiate between "boxes" of users. In their example it is the country USA, but if you define a "google user country" and let all users of google in it is ok to bundle the gbot with those. Grey area for sure, but might makes right.
seanp2k2 9y ago
Yep, that's literally https://en.m.wikipedia.org/wiki/Cloaking https://en.m.wikipedia.org/wiki/Cloaking . If they were a smaller site, they'd probably get outright banned by google.
snickerbockers 9y ago
There used to be a lot of sites that did that, and the result was that people would set their user-agents to match google's and get all the paywalled content for free.

The whole purpose of making a webcrawler for a search engine is that you are crawling the content that a user will see upon clicking a link in search results.

2_listerine_pls 9y ago
Even if it's behind a paywall, there is benefit in crawling it. For instance, if you happened to be a logged WSJ user, then google could show you the result based on your cookies.
- dragonwriter 9y ago
  Googlebot does not crawl the web separately for each user with a copy of that user's credentials borrowed from their browser. Aside from the privacy and security issues, it would require Google to multiply it's search resources by the number of users.
  - 2_listerine_pls 9y ago
    > Googlebot does not crawl the web separately for each user with a copy of that user's credentials borrowed from their browser. I did not say that. I said google could display the results to which you have access, based on your cookies. Assuming they had access to crawl the whole article.
    - CydeWeys 9y ago
      That would require Google to be in the business of knowing every site that you have a membership in. It is not possible for Google to get access to cookies for other sites, anyway. This is a pretty fundamentally important security restriction that browsers implement to protect you from nefarious sites. So it isn't possible for Google to know which sites you have paid accounts with unless you explicitly tell it.
- Flenser 9y ago
  > Even if it's behind a paywall, there is benefit in crawling it. How would that work? Would Google create or be given a login for WSJ? That would benefit WSJ as an incumbent news provider at the expense of startups too new or small to get special treatment from Google. How would a new website indicate to all search crawlers (so google doesn't befit at the expense of other search engines) how to get access to it's pay-for content in such a way that end-users can not also pretend to be seach crawlers and get access to the same content?

DougWebb 9y ago

Possible, but seems unlikely. To set that up the WSJ website would have to allow Googlebot access while denying others. Any filtering based on the url or HTTP headers would be discovered and abused by others. An approach based on a security token or IP filter could work, but would be un-managable on Google's side because of the scale of their spidering operation. It would be much more effective for them to use their position to force the WSJ to be an open website, or to accept that their paywalled content does not get indexed.

devicenull 9y ago
Just allow access from all of Google's IP space, as long as the user-agent contains "googlebot'. It's pretty trivial to do... Sure, some people could set up GCP instances and proxy it, but that's a very tiny percentage of people.
- DougWebb 9y ago
  It might start out as a tiny percentage, but all it takes is one person setting it up and letting the world know about it. Then pretty soon the WSJ is faced with millions of people getting free content again. They're complaining about people accessing their articles via a Google search and then clearing cookies to reset their counters. That's hardly mainstream; browsers have been burying the clear-cookies functionality deeper and deeper over the years because it's seen as an advanced-user-only kind of thing. And yet the WSJ has millions of people doing it, enough to make an impact on their bottom line.
  - devicenull 9y ago
    I'd bet it's browing in incognito mode versus actually clearing cookies.
- obstinate 9y ago
  Googlebot's IP range and GCP's IP ranges are disjoint.
- detaro 9y ago

hackcasual 9y ago

That's a no-no. https://support.google.com/webmasters/answer/66355?hl=en https://support.google.com/webmasters/answer/66355?hl=en

2_listerine_pls 9y ago
I went through the whole thing and it seems, from the video, that they could avoid cloaking by redirecting non-paid users to the sing-up page. This way WSJ would be showing the same (www.wsj.com/Bezos-with-hair) to every user. *based on the video
- jacquesm 9y ago
  > sing-up page. That's a new one. Buy a subscription to the WSJ now for a song.
- squeaky-clean 9y ago
  In the video, he stresses pretty hard that all forms of cloaking are disallowed, even if not malicious or deceiving. Unless the Googlebot has a paid subscription to WSJ that's still cloaking, as you're showing Googlebot a different page than a regular user. Google's rules and help documents are spread all over, but here's some from Google News about subscriptions: "If you prefer this option, please display a snippet of your article that is at least 80 words long and includes either an excerpt or a summary of the specific article. Since we do not permit "cloaking" -- the practice of showing Googlebot a full version of your article while showing users the subscription or registration version -- we will only crawl and display your content based on the article snippets you provide." edit, forgot link: https://support.google.com/news/publisher/answer/40543?hl=en https://support.google.com/news/publisher/answer/40543?hl=en
nso 9y ago
I guess what this one infact states is that what they were doing was okay. In the video the guy explains that while you are not allowed to treat gbots request in a way no other people are treated, it is okay to differentiate between "boxes" of users. In their example it is the country USA, but if you define a "google user country" and let all users of google in it is ok to bundle the gbot with those. Grey area for sure, but might makes right.

seanp2k2 9y ago

Yep, that's literally https://en.m.wikipedia.org/wiki/Cloaking https://en.m.wikipedia.org/wiki/Cloaking . If they were a smaller site, they'd probably get outright banned by google.

snickerbockers 9y ago

There used to be a lot of sites that did that, and the result was that people would set their user-agents to match google's and get all the paywalled content for free.