3 ms·
> Even if it's behind a paywall, there is benefit in crawling it. How would that work? Would Google create or be given a login for WSJ? That would benefit WSJ
by Flenser 9y ago
> Even if it's behind a paywall, there is benefit in crawling it.
How would that work? Would Google create or be given a login for WSJ? That would benefit WSJ as an incumbent news provider at the expense of startups too new or small to get special treatment from Google.
How would a new website indicate to all search crawlers (so google doesn't befit at the expense of other search engines) how to get access to it's pay-for content in such a way that end-users can not also pretend to be seach crawlers and get access to the same content?