4 ms·
Google in particular always uses an identifiable agent. So you could serve one set of content to Google and a separate set to actual visitors, preventing Google
by shaftway 5y ago
Google in particular always uses an identifiable agent. So you could serve one set of content to Google and a separate set to actual visitors, preventing Google from scraping the content.
They clearly want the benefit of being scraped (the traffic) without the cost (the loss of traffic from people who don't click past the headline to the article).