3 ms·
About ten years after creating Solr, Yonik Seeley joined Cloudera to work on integration between Apache Hadoop and Solr. There's an interesting connection here
by tomwheeler 5y ago
About ten years after creating Solr, Yonik Seeley joined Cloudera to work on integration between Apache Hadoop and Solr.
There's an interesting connection here: Doug Cutting, perhaps best known as the creator of Hadoop, was the Chief Architect at Cloudera. Most people recognize Doug as the creator of Apache Hadoop, but he also created Lucene. In fact, Hadoop originated from a Lucene subproject called Nutch, which aimed to build a scalable web crawler.
- ideonode 5y agoWasn't the inspiration for Hadoop not just a web crawler use-case, but also Google's famous MapReduce paper?
- tomwheeler 5y agoYes and no. The goal of the Nutch project was simply to create a web crawler, but it hit some scalability limits. Since Google had recently published two papers (MapReduce and Google Filesystem) that were quite relevant to scaling data processing and storage for a web crawler, Doug and Mike created an open source implementation of those ideas and redesigned the web crawler to use it. The technology had many applications beyond a web crawler, of course, but that was the original use case.