2 ms·
The big, important crawlers have known IPs don't they? Isn't that how news sites, quora and friends selectively apply their pay wall and still get indexed?
by morkalork 3y ago
The big, important crawlers have known IPs don't they? Isn't that how news sites, quora and friends selectively apply their pay wall and still get indexed?