4 ms·
If they are cloaking, can we get around the paywall by using the Google crawler user agent string?
by leephillips 6y ago
If they are cloaking, can we get around the paywall by using the Google crawler user agent string?
- tyingq 6y agoThat is one of many workarounds the various paywall-buster browser extensions use. Either setting to the Google crawler user-agent, or the Google AdBot agent. I would guess you would need to not send cookies also. They could also be clever and check that your IP/Netblock is a Google owned one.
- gpm 6y agoHuh, I thought they published a range of IP addresses they used to prevent this, but apparently they don't use an entirely consistent one and you need to do a dns request [1] to actually check if something is google's crawler. I'm willing to bet most organizations aren't doing that... so maybe. [1] https://developers.google.com/search/docs/advanced/crawling/verifying-googlebot https://developers.google.com/search/docs/advanced/crawling/...
- leephillips 6y agoI just installed a user agent switcher and tried it on a prominent financial news site. The offer to subscribe was replaced by the article when I reloaded using the Googlebot user agent.