3 ms·

Or possibly a common open resource from which services can pull regularly updated crawl data. I'm not sure if what I am thinking of exists or is feasible or not

Or possibly a common open resource from which services can pull regularly updated crawl data. I'm not sure if what I am thinking of exists or is feasible or not, just a thought from someone with little experience in crawling in general. Although, then that would put load on a single source and someone has to pay for those resources. Maybe if there was a distributed resource then, I wonder if block-chain technology or something similar factor into such a niche.

dredmorbius 8y ago
Right: that's "crawl, but share". Both models are fairly isomorphic, modulo the crawl / search locus. The key is on agreed data structures, practices, and access.
mindcrime 8y ago
That sounds a lot like what CommonCrawl does, and they are a good start. But from what I've seen, if you wanted to build a really competitive search engine, you'd still have to complement that with your own crawling. Not sure though, as I haven't done a deep dive into this topic.