3 ms·
>Many paywalled sites will let googlebot index their content but block other crawlers. Doesn't that infringe upon Google's own rules? I always thought Google d
by matheusmoreira 8y ago
>Many paywalled sites will let googlebot index their content but block other crawlers.
Doesn't that infringe upon Google's own rules? I always thought Google didn't like it when sites served its crawler content that's different from what users get when they follow Google's link.
- fauigerzigerk 8y agoThat's why many paywalled sites give you a few free articles per month if you're coming from a Google search results page. But it no longer works at all sites. Maybe the rule has been dropped now that paywalls are becoming more popular (with publishers that is)