3 ms·
Crawling is a challenge but not as much as mentioned. We have had a respectful bot since 2004, so the problems for us are more about awareness than hostility.
by ColinHayhurst 5y ago
Crawling is a challenge but not as much as mentioned. We have had a respectful bot since 2004, so the problems for us are more about awareness than hostility.
Indexing is the bigger challenge. An efficient fast service for an index of billions of pages is a different level to that for millions of pages.
- boyter 5y agoI have always found crawling harder personally. At least with indexing you can switch to batch processing, or split the index into real-time and stale portions to give that fresh feel. Just my personal opinion though. I have never had the chance to index billions of pages. Several hundred million is about as far as I have gone.