2 ms·
There's Common Crawl for the crawling aspect, about 3.2 billion pages last time I looked. One of the issues with that kind of detachment of jobs is crawl data f
by ricardo81 6y ago
There's Common Crawl for the crawling aspect, about 3.2 billion pages last time I looked. One of the issues with that kind of detachment of jobs is crawl data freshness.
- lambda_obrien 6y agoI'm thinking more like search indexers with like 100k pages, in a specialized category like 3d printing or basketball. I can index things like those on my home PC, practically.