3 ms·
Does it even make sense to serve different content to a bot than what a human would see? Isn't the search engine trying to rank content made for humans?
by XCSme 3y ago
Does it even make sense to serve different content to a bot than what a human would see? Isn't the search engine trying to rank content made for humans?
- Scarblac 3y agoIt makes sense if you know your content isn't nice for humans (e.g. full of ads and tracking stuff) but you want it to rank high anyway.
- pjc50 3y agoIt's an adversarial process. The search engine is, in theory, trying to rank by usefulness to the user, and the site owner is trying to maximize revenue by lying to the search engine. And the user.
- XCSme 3y agoThen, maybe the "crawler" should be an actual PC navigating to the browser, taking a screenshot (or live feed) of the page and processing it with AI.
- pjc50 3y agoEh, Google choose to be identifiable as googlebot and to obey robots.txt for other reasons of "good citizenship", because not everybody wants to be crawled.
- AlexandrB 3y agoI'm generally puzzled by Google's reluctance to do manual intervention in these cases. It's not like this is a secret. Just penalize the whole domain for 60 days every time a prominent site lies to the crawler.
- gtirloni 3y agoIf Google's support is any indication, Google doesn't like to involve humans in their processes. There probably isn't enough humans to do this manual intervention you propose.
- sulam 3y agoThere are very many sites where the content you see as a non-logged-in user is different from what you see if you have in your possession an all-important user cookie.