6 ms·
Yes, except we want a shared resource for an index so we aren't all crawling sites in excess. Eg, here's the latest central or distributed DB, the site in your
by cdnsteve 3y ago
Yes, except we want a shared resource for an index so we aren't all crawling sites in excess.
Eg, here's the latest central or distributed DB, the site in your personal list is on it so we don't need to send the crawler there because it has already been indexed and is crawled by someone else.