I'm hoping that someone sees value in creating and open source business around the CommonCrawl data, with a similar strategy as NodeJitsu, who opened source eno

I'm hoping that someone sees value in creating and open source business around the CommonCrawl data, with a similar strategy as NodeJitsu, who opened source enough of their work that others could build a parallel business to them. This approach is appropriate where the market will only exist if enough private actors realize that they need to cooperate for the market to exist at all. BitCoin is a perfect example of a similar ecosystem. It's a collective, distributed startup. Search tech beyond what Google is requires a similar cooperative effort. Right now the CommonCrawl data set is so big and the cost of setting up all the infrastructure to work on it is so expensive, that few people have the luxury to explore how search engines might work differently. I don't know how the search algo team does its job at Google, but it would be awesome if the interface made available to them was offered as a service that anyone could build a search business off of. That would be a great first step in allowing specialized search engines for specific verticals/domains to take off.