3 ms·
Now that sites are being more aggressive when detecting ad blockers, but they still want search engines to index the site and so they happily serve content to t
by czep 11y ago
Now that sites are being more aggressive when detecting ad blockers, but they still want search engines to index the site and so they happily serve content to the crawlers... Would it be possible to write a browser plugin that can let your session masquerade as a robot? Even pay walled sites let Google index their content, that's how they lure unsuspecting chumps to the site. So why couldn't we have a plugin to imitate a robot, getting the content and by having to suffer through several megabytes of ad network code?
- jnky 11y agoSure, there are browser plugins to override the HTTP user agent header and in my experience impersonating Google Bot works quite well. On the other hand, this tactic runs afoul with sites detecting your UA for browser specific hacks (especially with HTML5 video) or parts of a sites where bots are blocked by policy. Also the IP ranges of search bots are more or less well known, so it would be trivial for publishers to block this kind of hack, if they wanted.
- tomjen3 11y agoBy googles cloaking rules they are not allowed to serve different content, so presumably just turning of javascript would work just as well.