3 ms·

You can't do it client side. Well-implemented paywalls clip the content server-side. If you're Google, you get the full HTML, if you're not, you don't.

by Swizec 5y ago

isiahl 5y ago
But I could change my clients User Agent to match the Google Crawler?
- lmm 5y ago
  Sure, and then you've got a cat-and-mouse game with all the other ways that sites can fingerprint you.
- dreamcompiler 5y ago
  Nope. Because your IP address is not one of Google's. This trick worked years ago but not today.
  - deleted 5y ago
    [deleted]
  - donkarma 5y ago
    use a GCP instance
    - eminence32 5y ago
      I'm pretty sure that CGP instances don't allocate IP addresses from the same range as google crawlers. See also: https://developers.google.com/search/docs/advanced/crawling/verifying-googlebot https://developers.google.com/search/docs/advanced/crawling/...
PragmaticPulp 5y ago
> If you're Google, you get the full HTML, if you're not, you don't. reply This company isn't Google either, though. Or are you suggesting that they're using Google Compute Engine and therefore get the unpaywalled version?
- gruez 5y ago
  That might work with crappy implementations (eg. they check ASN = Google), but google publishes a whitelist of their crawler bot IP ranges, so spoofing that is all but impossible.
makeworld 5y ago
This is false. I use this extension every day and it works great. There are many ways to bypass paywalls. For example, you can pretend to be Google. https://gitlab.com/magnolia1234/bypass-paywalls-chrome-clean https://gitlab.com/magnolia1234/bypass-paywalls-chrome-clean
imgabe 5y ago
Doesn't Google penalize sites that show something different to the crawler than they do to a regular user?
- probably_wrong 5y ago
  I know that this used to be common knowledge, but evidence points to "no". Or at least to "either not anymore or not strongly enough". Clear examples of websites that block content from me: Pinterest, Facebook, LinkedIn, newspapers.
  - JimDabell 5y ago
    According to Google: > Cloaking refers to the practice of presenting different content or URLs to human users and search engines. Cloaking is considered a violation of Google's Webmaster Guidelines because it provides our users with different results than they expected. — https://developers.google.com/search/docs/advanced/guidelines/cloaking https://developers.google.com/search/docs/advanced/guideline... It’s clear that Google aren’t enforcing this rule thoroughly though. Edit: It’s possible that the allowed sites are using JSON-LD to avoid being classified as cloaking: > This page describes how to use schema.org JSON-LD to indicate paywalled content on your site with CreativeWork properties. This structured data helps Google differentiate paywalled content from the practice of cloaking, which violates our guidelines. — https://developers.google.com/search/docs/advanced/structured-data/paywalled-content https://developers.google.com/search/docs/advanced/structure...

You can't do it client side. Well-implemented paywalls clip the content server-side. If you're Google, you get the full HTML, if you're not, you don't.

by Swizec 5y ago

You can't do it client side. Well-implemented paywalls clip the content server-side. If you're Google, you get the full HTML, if you're not, you don't.

isiahl 5y ago

But I could change my clients User Agent to match the Google Crawler?

lmm 5y ago
Sure, and then you've got a cat-and-mouse game with all the other ways that sites can fingerprint you.
dreamcompiler 5y ago
Nope. Because your IP address is not one of Google's. This trick worked years ago but not today.
- deleted 5y ago
  [deleted]
- donkarma 5y ago
  use a GCP instance
  - eminence32 5y ago
    I'm pretty sure that CGP instances don't allocate IP addresses from the same range as google crawlers. See also: https://developers.google.com/search/docs/advanced/crawling/verifying-googlebot https://developers.google.com/search/docs/advanced/crawling/...

PragmaticPulp 5y ago

> If you're Google, you get the full HTML, if you're not, you don't. reply This company isn't Google either, though. Or are you suggesting that they're using Google Compute Engine and therefore get the unpaywalled version?

gruez 5y ago
That might work with crappy implementations (eg. they check ASN = Google), but google publishes a whitelist of their crawler bot IP ranges, so spoofing that is all but impossible.

makeworld 5y ago

This is false. I use this extension every day and it works great. There are many ways to bypass paywalls. For example, you can pretend to be Google. https://gitlab.com/magnolia1234/bypass-paywalls-chrome-clean https://gitlab.com/magnolia1234/bypass-paywalls-chrome-clean

imgabe 5y ago

Doesn't Google penalize sites that show something different to the crawler than they do to a regular user?

probably_wrong 5y ago
I know that this used to be common knowledge, but evidence points to "no". Or at least to "either not anymore or not strongly enough". Clear examples of websites that block content from me: Pinterest, Facebook, LinkedIn, newspapers.
- JimDabell 5y ago
  According to Google: > Cloaking refers to the practice of presenting different content or URLs to human users and search engines. Cloaking is considered a violation of Google's Webmaster Guidelines because it provides our users with different results than they expected. — https://developers.google.com/search/docs/advanced/guidelines/cloaking https://developers.google.com/search/docs/advanced/guideline... It’s clear that Google aren’t enforcing this rule thoroughly though. Edit: It’s possible that the allowed sites are using JSON-LD to avoid being classified as cloaking: > This page describes how to use schema.org JSON-LD to indicate paywalled content on your site with CreativeWork properties. This structured data helps Google differentiate paywalled content from the practice of cloaking, which violates our guidelines. — https://developers.google.com/search/docs/advanced/structured-data/paywalled-content https://developers.google.com/search/docs/advanced/structure...