2 ms·
There are two problems with this. First, they don't want to. In fact, if a search engine can figure out that a link is going to lead to a paywall, they'll prob
by AnthonyMouse 11y ago
There are two problems with this.
First, they don't want to. In fact, if a search engine can figure out that a link is going to lead to a paywall, they'll probably want to reduce the ranking of the result, because the user is not going to want results they can't actually look at.
Second, it would be a massive antitrust violation because it would prevent access by competing crawlers. The only way around that is to allow access to anyone who claims they're a crawler, which was the original problem.
- LunaSea 11y agoThe current situation with the WSJ could already be considered an antitrust violation. It's whitelisting one crawler and leaving the other ones out.