4 ms·
I suspect the paywall protected articles that Google runs across aren't protected by IP address, but by User-Agent string. I've seen a few hacks for this (look
by Travis 18y ago
I suspect the paywall protected articles that Google runs across aren't protected by IP address, but by User-Agent string. I've seen a few hacks for this (look at the NYT User Agent hack), and the FF plugin that allows you to alter your UA string indicates this similarly.
Put another way, I don't know if Google necessarily has registered all of its crawling IP addresses. Sure, you could do a reverse IP lookup -- but you'd have to do that on every request, which can get expensive.
No, the cheap-easy route is probably something as mundane as UA string.
- asciilifeform 18y agoFaking the UA string does nothing on all of the journal sites I've tried.
- rjprins 18y agoThere are not that many publishers and not all crawlers need access. My guess would be the publishers gave Google their own license with which a specific crawler could identify itself.