3 ms·
I understand they're having to pay for access to crawler results, to avoid the cost of indexing and crawling the entire web (though they do some themselves).
by g_p 4y ago
I understand they're having to pay for access to crawler results, to avoid the cost of indexing and crawling the entire web (though they do some themselves).
- worldofmatthew 4y agoWould it very quickly at those costs become cheaper for them to crawl the most popular 10 million websites themselves and only use external providers as backfill?