4 ms·
I've been thinking about this for a while. It's infeasible for any one person to run a Google-scale web crawler -- but if millions of people join in? I've used
by andai 6y ago
I've been thinking about this for a while. It's infeasible for any one person to run a Google-scale web crawler -- but if millions of people join in?
I've used file sharing software which has the same "tree style" query forwarding. I think latency would be the big issue though, we're used to getting a result instantly -- and the right result too (Google does a lot of extra work knowing what you actually mean, not just keyword search).
- mark_l_watson 6y agoCommon Crawl is a good resource if having crawl data potentially being from a month ago is OK for your application. Donate and use their crawl data.