10 ms·

If you click a search result and end up seeing something completely different than what you expected based on the search result snippet, it shouldn't matter if

If you click a search result and end up seeing something completely different than what you expected based on the search result snippet, it shouldn't matter if you're the WSJ or a scam site trying to hack your Google rank. It's deceiving the user and inflating your search ranking at the expense of more deserving listings.

eropple 9y ago
Yup. We used to call it "cloaking", I don't know if that's still a term in use in the SEO world.
- tyingq 9y ago
  Google still calls it that. https://support.google.com/webmasters/answer/66355?hl=en https://support.google.com/webmasters/answer/66355?hl=en
  - artursapek 9y ago
    They are known to crawl using human-like user agents instead of the typical Googlebot one precisely to counter this (weak) effort at playing the system. I'm surprised WSJ is surprised by the outcome here.
    - x0x0 9y ago
      They're in no way surprised; they're trying to marshal public outrage or lean on anti-trust efforts in order to force google to rank them higher.
  - mistermann 9y ago
    I wish google would give us somewhere to report sites that are doing this.
    - tyingq 9y ago
      It's here: https://www.google.com/webmasters/tools/spamreportform?hl=en https://www.google.com/webmasters/tools/spamreportform?hl=en It specifically mentions cloaking.
    - daddyo 9y ago
      https://www.google.com/webmasters/tools/spamreportform?hl=en https://www.google.com/webmasters/tools/spamreportform?hl=en > Webspam pages try to get better placement in Google's search results by using various tricks such as hidden text, doorway pages, cloaking, or sneaky redirects. These techniques attempt to compromise the quality of our results and degrade the search experience for everyone.
      - sogen 9y ago
        According to this Facebook should rank very low. Facebook doesn't even need Google, their users just visit the site directly. I guess Mark zuckerberg doesn't lose sleep over this.
        daddyo 9y ago
        Google does not penalize Facebook for web spam. Facebook, to my knowledge, does not do: - hidden text, - doorway pages, - cloaking, or - sneaky redirects They just show a big popover nagging you to log-in. But you can click this away. If certain Facebook content pages rank low, or do not rank at all, it is because Facebook actively blocks Googlebot from accessing the content, not because Facebook is trying to deceive Google (or the user). Though Facebook does not need Google, it could get quite a lot more visitors if it lowered the wall of its garden a bit. As is, Facebook is an inaccessible social echo chamber, and I don't lose any sleep over this.
seanp2k2 9y ago
This. I installed the personal blacklist extension for Chrome just to blacklist Quora results because of this. I wish Google would actually punish them more. I click through on something which looks relevant, and the result is a useless login screen. LinkedIn is the same.
- pkkm 9y ago
  You can bypass the Quora login screen by appending ?share=1 to the URL. There's also an userscript which does this automatically: https://github.com/sindresorhus/quora-unblocker-userscript https://github.com/sindresorhus/quora-unblocker-userscript
  - kevmo 9y ago
    Ah, I made a similar (but simpler) one years ago: https://github.com/kevmo/greasemonkey/blob/master/quora_upvotes.user.js https://github.com/kevmo/greasemonkey/blob/master/quora_upvo...
  - devrandomguy 9y ago
    If you change your browser's UserAgent string to Googlebot, then your client will be treated as a first-class citizen, by many of these sites. Google always wins, so let's all be Google.
    - Xeoncross 9y ago
      Works great until you stumble on one of the big sites who will auto-ban you for not having a valid Google IP address.
      - devrandomguy 9y ago
        Is that really a thing? That must be such a hazard for their developers. I usually have a test for sites that I work on, that scrapes a few URLs as Googlebot, to verify that they are getting an optimized view (no JS, structural-only css).
danso 9y ago
My thoughts exactly. If WSJ wants to increase monetization through paywalls, it has every right to do so, but should suffer in SERP accordingly as a paywalled-article is generally not what users are looking for.
amenod 9y ago
I think this is not what WSJ is after. I think they are trying to pressure Google to find a more suitable policy (suitable to WSJ) by putting Google users behind paywall while allowing free visits from social media. They are probably laying the groundwork for future battles, sorry, I meant talks.
siliconc0w 9y ago
+1, this is already pretty bad for a lot of searches with so many sites erecting paywalls, pushing required logins, or anti-ad-block. Not sure if they track this but whatever the Googlebot's view of the site's content, if it's constantly bouncing users back to the search page it should get hit with a hard penalty.
seshagiric 9y ago
Why can't WSJ provide full indexing access to Google crawler robot but keep it paid for users? That way at least the scam problem will be solved?
- CobrastanJorji 9y ago
  They'd probably be happy to do that, but my understanding is that Google generally frowns strongly upon sites that display different content (or no content) to users when they tell Google those pages have content. I could imagine Google adding some sort of "content is locked behind a paywall" indicator on search results, but if I'm searching for something on the web, a link to blocked content is not very helpful most of the time.
  - yomly 9y ago
    Yep - imagine a world where most the top search results are subscription only. That's a terrible UX for a Google searcher to have to go past page 1 to actually have content. Google definitely doesn't want that, and its also anti-open web
    - dqv 9y ago
      I actually would find it useful if Google had an indicator like 3 pay-walled results omitted, show them? At least being able to know it exists can help consumers decide if they should pay for an article/subscription.
      - sqeaky 9y ago
        Google could charge paywalled sites to make such a feature, but as a user I never want to see that.
      - tripzilch 9y ago
        if Google is going to accommodate pay-walled results like that, you can expect this indicator won't just show/hide 3 high quality pay-walled results, but also 30 shitty ones that will want to get in on the potential pay-wall business.
sgustard 9y ago
If Google doesn't differentiate between "the WSJ or a scam site", its value to me is severely curtailed.
- rblatz 9y ago
  Then the WSJ shouldn't cloak content like a scam site.
- dragonwriter 9y ago
  Since WSJ objectively is a scam site in the relevant dimension, offering different cotbent to Googlebot than to search users, it absolutely should be treated exactly like other scam sites.
  - deleted 9y ago
    [deleted]
  - krschultz 9y ago
    Different content to some search users. All WSJ subscribers see the same thing as the Googlebot.
    - dragonwriter 9y ago
      > Different content to some search users. All WSJ subscribers see the same thing as the Googlebot. WSJ subscribers, online and print combined, are in the low single digit millions. Google search monthly unique users are about three orders of magnitude greater. WSJ online subscribers are close enough to 0% of Google's users as to make no difference. So, that "some" is essentially all.
      - buxtehude 9y ago
        The vast majority of google searches are certainly not all searching for things that would lead them to WSJ. If we take just the population of the US in 2017 (326.5M), and assume every American searches via Google for their news, we're looking at give or take 1% of the US with the WSJ subscriber estimate you provided (~3M). We can refine these numbers further... 19.4% of the US population is less than or equal to 14 years of age (18 would be better, but couldn't find) - so that gives 263M potential American news readers The Pew Research Center http://www.journalism.org/2016/07/07/pathways-to-news/ http://www.journalism.org/2016/07/07/pathways-to-news/ states that about 38% of adults in 2016 often get news online (~99M) That leads to at least 3% of US Google news searchers are potential WSJ subscribers. So that's not a tiny number (yes the number could be adjusted for worldwide English speaking news googlers - but I think I've made my point). How many of these subscribers are wealthy and coveted by advertisers? What if more news publishers follow WSJ and you happen to be a subscriber of that content? With the amount of information that Google has on its users, I don't see why it can't adjust search results based on whether or not you subscribe - and bring value to whichever side of the paywall you reside on.
daxfohl 9y ago
This actually sounds like a really interesting/profitable problem to solve. I'm sure there are tons of sites that want to be pay sites, but also want articles to show up in google search results. But users don't want to get inaccessible results. Google (or whomever) needs some solution to handle non-free results intelligently. Allow users to filter out non-free results, to configure which journals they do have subscriptions to, etc. Even an easy way to make micropayments. If Steve Jobs was still alive, I'd bet Apple would be working on a competing search engine with some of these features.
- yomly 9y ago
  That is a step that tends towards a walled-in closed Internet...
  - daxfohl 9y ago
    It's probably inevitable. TV was free airwaves for so long. Now it's cable subscriptions. (And there's still advertising). Internet is already controlled by your ISP, so I have a hard time seeing that being "open" for much longer. Browsers, content providers, etc will all be regulated / commercialized / sandboxed. Free zero-revenue startup idea: there'll be an IP-over-ham-radio or something to preserve "internet classic". (Largest use will be bitcoin-for-pornography).
- Nition 9y ago
  I agree. Advertising is not a great business model on its own, particularly with the proliferation of adblockers. If there was a way to get people supporting more paid content then IMO the quality of content could also improve. But who knows how all this will go in the end. There's always been a lot of high quality free content on the Internet as well, just because people are willing to share their knowledge with each other. krschultz's comment is really relevant as well.[1] In a complete system, search should actually know about what you subscribe to already, and not penalise those results for you. [1] https://news.ycombinator.com/item?id=14492870 https://news.ycombinator.com/item?id=14492870
  - 9y ago
abandonliberty 9y ago
You're right, this is wrong in our current paradigm. But our current paradigm is wrong, too. I want a world that supports business models other than web advertising. It negatively impacts journalistic integrity and freedom while further exacerbating the race to the bottom search engines and other content aggregators create. WSJ is responding poorly to a bad situation. I suspect it'll cost them. http://brave.com/ http://brave.com/ is one approach.
- pgwhalen 9y ago
  I'm totally with you, but what would you propose as a pragmatic solution? I really can't think of anything myself.
- scott_karana 9y ago
  Out of curiosity, do you think journalistic integrity was also impacted when newspapers used traditional paper advertisements as their source of revenue? Eg, have newspapers _ever_ had integrity, and if they _did_, what's different?
  - chii 9y ago
    The papers lost classified ads. They were a large revenue source which did not dictate an agenda.
  - dragonwriter 9y ago
    > Eg, have newspapers _ever_ had integrity, and if they _did_, what's different? They did somewhat before the massive corporate consolidation starting in, IIRC, the late 1970s when newsrooms started getting axes and the major dailies progressively became skins over wire services and lightly-rewritten press releases. The internet often gets the blame, but it providing actual competition was decades after the terminal quality and subscribership decline of American newspapers began. It actually was the internet competition (both wire services being directly available to readers and the loss of advertising) that actually got some of them talking about building up newsrooms, rebooting investigative journalism, and relying more on subscription income (paid subscription was never paying the bills before, it was pursued as a key metric advertisers used in determining how much it was worth to advertise in a paper.)
krschultz 9y ago
But many Google users do see the content. I wish Google knew that I paid for subscriptions to WaPo, WSJ, & NYT and ranked things accordingly. I never want to open an ad supported story when the same thing is covered by a publisher I subscribe to.
- jcranberry 9y ago
  I think it's not far fetched to imagine someone with a paid subscription is subscribing to use it habitually as opposed to just having access whenever they clicked on a link via google.
  - krschultz 9y ago
    Right but lets say I search for an old topic (not current news). It won't be on the home page of the WSJ, NYTimes, or WaPo. But I'd like Google to surface that for me since I'm a subscriber and pay to have access to journalism like that.
eveningcoffee 9y ago
WSJ can in principle counter it by offering a summary for the unregistered logins (and bots).
rhino369 9y ago
It isn't necessarily deceiving. In many situations, I'd rather know where the information is rather than Google pretend it doesn't exist. For example, Google scholar will search papers that are behind a paywall. By blocking them, I wouldn't even know what to purchase. This is google foisting their business model down our throats.
downandout 9y ago
100% agree. I've actually reported WSJ to Google several times for cloaking (that's what Google calls this bait-and-switch). The penalty for this is supposed to be deindexing of the entire site from all Google results, but apparently WSJ gets a pass on that. At least they're being punished to some degree. For the record, if anybody needs a draggable WSJ paywall bypass bookmarklet, I put one up here: http://salzeko.com/wsj/ http://salzeko.com/wsj/
Bartweiss 9y ago
Agreed. There's an interesting difference between overlay paywalls like Wired uses, and content-not-loaded walls like WSJ uses. In the Wired case, they text is sent to you but they try to stop you from reading it. In the WSJ case, they don't even send you the text of the page you supposedly clicked on. Since we're in the second case, this isn't even a decision by Google. The WSJ actually isn't sending you the data in the search result snippet, so the crawler rightly says "wow, nothing useful here". The complex, ideal solution might let me tell Google "search as though I'm a WSJ member", but short of that they're accurately assessing what content is actually available.

eropple 9y ago

Yup. We used to call it "cloaking", I don't know if that's still a term in use in the SEO world.

tyingq 9y ago
Google still calls it that. https://support.google.com/webmasters/answer/66355?hl=en https://support.google.com/webmasters/answer/66355?hl=en
- artursapek 9y ago
  They are known to crawl using human-like user agents instead of the typical Googlebot one precisely to counter this (weak) effort at playing the system. I'm surprised WSJ is surprised by the outcome here.
  - x0x0 9y ago
    They're in no way surprised; they're trying to marshal public outrage or lean on anti-trust efforts in order to force google to rank them higher.
- mistermann 9y ago
  I wish google would give us somewhere to report sites that are doing this.
  - tyingq 9y ago
    It's here: https://www.google.com/webmasters/tools/spamreportform?hl=en https://www.google.com/webmasters/tools/spamreportform?hl=en It specifically mentions cloaking.
  - daddyo 9y ago
    https://www.google.com/webmasters/tools/spamreportform?hl=en https://www.google.com/webmasters/tools/spamreportform?hl=en > Webspam pages try to get better placement in Google's search results by using various tricks such as hidden text, doorway pages, cloaking, or sneaky redirects. These techniques attempt to compromise the quality of our results and degrade the search experience for everyone.
    - sogen 9y ago
      According to this Facebook should rank very low. Facebook doesn't even need Google, their users just visit the site directly. I guess Mark zuckerberg doesn't lose sleep over this.
      - daddyo 9y ago
        Google does not penalize Facebook for web spam. Facebook, to my knowledge, does not do: - hidden text, - doorway pages, - cloaking, or - sneaky redirects They just show a big popover nagging you to log-in. But you can click this away. If certain Facebook content pages rank low, or do not rank at all, it is because Facebook actively blocks Googlebot from accessing the content, not because Facebook is trying to deceive Google (or the user). Though Facebook does not need Google, it could get quite a lot more visitors if it lowered the wall of its garden a bit. As is, Facebook is an inaccessible social echo chamber, and I don't lose any sleep over this.

seanp2k2 9y ago

This. I installed the personal blacklist extension for Chrome just to blacklist Quora results because of this. I wish Google would actually punish them more. I click through on something which looks relevant, and the result is a useless login screen. LinkedIn is the same.

pkkm 9y ago
You can bypass the Quora login screen by appending ?share=1 to the URL. There's also an userscript which does this automatically: https://github.com/sindresorhus/quora-unblocker-userscript https://github.com/sindresorhus/quora-unblocker-userscript
- kevmo 9y ago
  Ah, I made a similar (but simpler) one years ago: https://github.com/kevmo/greasemonkey/blob/master/quora_upvotes.user.js https://github.com/kevmo/greasemonkey/blob/master/quora_upvo...
- devrandomguy 9y ago
  If you change your browser's UserAgent string to Googlebot, then your client will be treated as a first-class citizen, by many of these sites. Google always wins, so let's all be Google.
  - Xeoncross 9y ago
    Works great until you stumble on one of the big sites who will auto-ban you for not having a valid Google IP address.
    - devrandomguy 9y ago
      Is that really a thing? That must be such a hazard for their developers. I usually have a test for sites that I work on, that scrapes a few URLs as Googlebot, to verify that they are getting an optimized view (no JS, structural-only css).

danso 9y ago

My thoughts exactly. If WSJ wants to increase monetization through paywalls, it has every right to do so, but should suffer in SERP accordingly as a paywalled-article is generally not what users are looking for.

amenod 9y ago

I think this is not what WSJ is after. I think they are trying to pressure Google to find a more suitable policy (suitable to WSJ) by putting Google users behind paywall while allowing free visits from social media. They are probably laying the groundwork for future battles, sorry, I meant talks.

siliconc0w 9y ago

+1, this is already pretty bad for a lot of searches with so many sites erecting paywalls, pushing required logins, or anti-ad-block. Not sure if they track this but whatever the Googlebot's view of the site's content, if it's constantly bouncing users back to the search page it should get hit with a hard penalty.

seshagiric 9y ago

Why can't WSJ provide full indexing access to Google crawler robot but keep it paid for users? That way at least the scam problem will be solved?

CobrastanJorji 9y ago
They'd probably be happy to do that, but my understanding is that Google generally frowns strongly upon sites that display different content (or no content) to users when they tell Google those pages have content. I could imagine Google adding some sort of "content is locked behind a paywall" indicator on search results, but if I'm searching for something on the web, a link to blocked content is not very helpful most of the time.
- yomly 9y ago
  Yep - imagine a world where most the top search results are subscription only. That's a terrible UX for a Google searcher to have to go past page 1 to actually have content. Google definitely doesn't want that, and its also anti-open web
  - dqv 9y ago
    I actually would find it useful if Google had an indicator like 3 pay-walled results omitted, show them? At least being able to know it exists can help consumers decide if they should pay for an article/subscription.
    - sqeaky 9y ago
      Google could charge paywalled sites to make such a feature, but as a user I never want to see that.
    - tripzilch 9y ago
      if Google is going to accommodate pay-walled results like that, you can expect this indicator won't just show/hide 3 high quality pay-walled results, but also 30 shitty ones that will want to get in on the potential pay-wall business.

sgustard 9y ago

If Google doesn't differentiate between "the WSJ or a scam site", its value to me is severely curtailed.

rblatz 9y ago
Then the WSJ shouldn't cloak content like a scam site.
dragonwriter 9y ago
Since WSJ objectively is a scam site in the relevant dimension, offering different cotbent to Googlebot than to search users, it absolutely should be treated exactly like other scam sites.
- deleted 9y ago
  [deleted]
- krschultz 9y ago
  Different content to some search users. All WSJ subscribers see the same thing as the Googlebot.
  - dragonwriter 9y ago
    > Different content to some search users. All WSJ subscribers see the same thing as the Googlebot. WSJ subscribers, online and print combined, are in the low single digit millions. Google search monthly unique users are about three orders of magnitude greater. WSJ online subscribers are close enough to 0% of Google's users as to make no difference. So, that "some" is essentially all.
    - buxtehude 9y ago
      The vast majority of google searches are certainly not all searching for things that would lead them to WSJ. If we take just the population of the US in 2017 (326.5M), and assume every American searches via Google for their news, we're looking at give or take 1% of the US with the WSJ subscriber estimate you provided (~3M). We can refine these numbers further... 19.4% of the US population is less than or equal to 14 years of age (18 would be better, but couldn't find) - so that gives 263M potential American news readers The Pew Research Center http://www.journalism.org/2016/07/07/pathways-to-news/ http://www.journalism.org/2016/07/07/pathways-to-news/ states that about 38% of adults in 2016 often get news online (~99M) That leads to at least 3% of US Google news searchers are potential WSJ subscribers. So that's not a tiny number (yes the number could be adjusted for worldwide English speaking news googlers - but I think I've made my point). How many of these subscribers are wealthy and coveted by advertisers? What if more news publishers follow WSJ and you happen to be a subscriber of that content? With the amount of information that Google has on its users, I don't see why it can't adjust search results based on whether or not you subscribe - and bring value to whichever side of the paywall you reside on.

daxfohl 9y ago

This actually sounds like a really interesting/profitable problem to solve. I'm sure there are tons of sites that want to be pay sites, but also want articles to show up in google search results. But users don't want to get inaccessible results. Google (or whomever) needs some solution to handle non-free results intelligently. Allow users to filter out non-free results, to configure which journals they do have subscriptions to, etc. Even an easy way to make micropayments. If Steve Jobs was still alive, I'd bet Apple would be working on a competing search engine with some of these features.

yomly 9y ago
That is a step that tends towards a walled-in closed Internet...
- daxfohl 9y ago
  It's probably inevitable. TV was free airwaves for so long. Now it's cable subscriptions. (And there's still advertising). Internet is already controlled by your ISP, so I have a hard time seeing that being "open" for much longer. Browsers, content providers, etc will all be regulated / commercialized / sandboxed. Free zero-revenue startup idea: there'll be an IP-over-ham-radio or something to preserve "internet classic". (Largest use will be bitcoin-for-pornography).
Nition 9y ago
I agree. Advertising is not a great business model on its own, particularly with the proliferation of adblockers. If there was a way to get people supporting more paid content then IMO the quality of content could also improve. But who knows how all this will go in the end. There's always been a lot of high quality free content on the Internet as well, just because people are willing to share their knowledge with each other. krschultz's comment is really relevant as well.[1] In a complete system, search should actually know about what you subscribe to already, and not penalise those results for you. [1] https://news.ycombinator.com/item?id=14492870 https://news.ycombinator.com/item?id=14492870
- 9y ago

abandonliberty 9y ago

You're right, this is wrong in our current paradigm. But our current paradigm is wrong, too. I want a world that supports business models other than web advertising. It negatively impacts journalistic integrity and freedom while further exacerbating the race to the bottom search engines and other content aggregators create. WSJ is responding poorly to a bad situation. I suspect it'll cost them. http://brave.com/ http://brave.com/ is one approach.

pgwhalen 9y ago
I'm totally with you, but what would you propose as a pragmatic solution? I really can't think of anything myself.
scott_karana 9y ago
Out of curiosity, do you think journalistic integrity was also impacted when newspapers used traditional paper advertisements as their source of revenue? Eg, have newspapers _ever_ had integrity, and if they _did_, what's different?
- chii 9y ago
  The papers lost classified ads. They were a large revenue source which did not dictate an agenda.
- dragonwriter 9y ago
  > Eg, have newspapers _ever_ had integrity, and if they _did_, what's different? They did somewhat before the massive corporate consolidation starting in, IIRC, the late 1970s when newsrooms started getting axes and the major dailies progressively became skins over wire services and lightly-rewritten press releases. The internet often gets the blame, but it providing actual competition was decades after the terminal quality and subscribership decline of American newspapers began. It actually was the internet competition (both wire services being directly available to readers and the loss of advertising) that actually got some of them talking about building up newsrooms, rebooting investigative journalism, and relying more on subscription income (paid subscription was never paying the bills before, it was pursued as a key metric advertisers used in determining how much it was worth to advertise in a paper.)

krschultz 9y ago

But many Google users do see the content. I wish Google knew that I paid for subscriptions to WaPo, WSJ, & NYT and ranked things accordingly. I never want to open an ad supported story when the same thing is covered by a publisher I subscribe to.

jcranberry 9y ago
I think it's not far fetched to imagine someone with a paid subscription is subscribing to use it habitually as opposed to just having access whenever they clicked on a link via google.
- krschultz 9y ago
  Right but lets say I search for an old topic (not current news). It won't be on the home page of the WSJ, NYTimes, or WaPo. But I'd like Google to surface that for me since I'm a subscriber and pay to have access to journalism like that.

eveningcoffee 9y ago

WSJ can in principle counter it by offering a summary for the unregistered logins (and bots).

rhino369 9y ago

It isn't necessarily deceiving. In many situations, I'd rather know where the information is rather than Google pretend it doesn't exist. For example, Google scholar will search papers that are behind a paywall. By blocking them, I wouldn't even know what to purchase. This is google foisting their business model down our throats.

downandout 9y ago

100% agree. I've actually reported WSJ to Google several times for cloaking (that's what Google calls this bait-and-switch). The penalty for this is supposed to be deindexing of the entire site from all Google results, but apparently WSJ gets a pass on that. At least they're being punished to some degree. For the record, if anybody needs a draggable WSJ paywall bypass bookmarklet, I put one up here: http://salzeko.com/wsj/ http://salzeko.com/wsj/

Bartweiss 9y ago

Agreed. There's an interesting difference between overlay paywalls like Wired uses, and content-not-loaded walls like WSJ uses. In the Wired case, they text is sent to you but they try to stop you from reading it. In the WSJ case, they don't even send you the text of the page you supposedly clicked on. Since we're in the second case, this isn't even a decision by Google. The WSJ actually isn't sending you the data in the search result snippet, so the crawler rightly says "wow, nothing useful here". The complex, ideal solution might let me tell Google "search as though I'm a WSJ member", but short of that they're accurately assessing what content is actually available.