5 ms·
Makes me wonder if creating a plugin that makes your browser pretend to be the Google indexing bot would give you secret access to all paysites?
by meghan_rain 4y ago
Makes me wonder if creating a plugin that makes your browser pretend to be the Google indexing bot would give you secret access to all paysites?
- jffry 4y agoCertainly at this point anybody serious about wanting to give Google special access through their paywall would allow based on the published IP blocks [1] and not an easily spoofed UA header [1] https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot#use-automatic-solutions https://developers.google.com/search/docs/crawling-indexing/...
- brookst 4y agoI remember the good old days when Google penalized sites for showing content to their crawler that wasn't available to normal users.
- jonas-w 4y agohttps://12ft.io https://12ft.io works on some sites through pretending to be the google bot. You can also access any site in the google cache with prepending "https://webcache.googleusercontent.com/search?q=cache https://webcache.googleusercontent.com/search?q=cache:", that will you show you the website like the google bot saw it. For example github.com would become "https://webcache.googleusercontent.com/search?q=cache:https%3A%2F%2Fgithub.com%2F https://webcache.googleusercontent.com/search?q=cache:https%..." It is still worth to try, but many sites already prevent this.
- stefandesu 4y agoIt just recently stopped working for Zeit.de articles. Seems like their paywall is now higher than 12 feet...