4 ms·
The scraper bots are exactly why they're making the full information available to non-logged in users, so it shows up in Google's index. I believe it's consider
by tomkarlo 12y ago
The scraper bots are exactly why they're making the full information available to non-logged in users, so it shows up in Google's index. I believe it's considered a no-no to show the bot a different page from a non-logged-in human viewer, so they don't really have a choice.
Hiding it from users with login/session cookies, however, is just being annoying and depending on the fact that most people won't know enough to switch to an incognito window.