4 ms·
WSJ does the same. It's simply done by detecting user-agent during the crawl, and then referrer during regular use.
by timcederman 18y ago
WSJ does the same. It's simply done by detecting user-agent during the crawl, and then referrer during regular use.
- Hexstream 18y agoAre you sure? I've read in the Google policies that you're supposed to serve the same content to Googlebot as your users or you're penalized in PageRank.
- timcederman 18y agoCorrect. Technically in this case they are in the clear because what people see on Google is what they will get when they click on the link. What is currently penalized is when you do this without checking the referrer and show everyone a sign-up screen. That'll get you penalized in a serious hurry.