3 ms·
Can you emulate googles crawler to trick your way around paywalls?
by ykevinator 7y ago
Can you emulate googles crawler to trick your way around paywalls?
- lucb1e 7y agoOr google translate to trick your way around IP whitelists? (Or maybe even through a proxy on a Google VPS?)
- rednixion 7y agoSetting a Googlebot UA will work for any low tech filter, besides looking at the ASN of the source IP or the javascript engine behavior I'm not sure what else sites could really do to figure it out sans coordinating with Google to define the indexer source(browsing/click pattern maybe). There's bound to be at least one googler in this thread and since the ASN check would be the hardest to fake and easiest to setup... someone go change their UA and go to nytimes or something from the office and see if it works(bonus points for any SRE who does it from a data center).