2 ms·

Wasn't the inspiration for Hadoop not just a web crawler use-case, but also Google's famous MapReduce paper?

• tomwheeler 5y ago
  Yes and no. The goal of the Nutch project was simply to create a web crawler, but it hit some scalability limits. Since Google had recently published two papers (MapReduce and Google Filesystem) that were quite relevant to scaling data processing and storage for a web crawler, Doug and Mike created an open source implementation of those ideas and redesigned the web crawler to use it. The technology had many applications beyond a web crawler, of course, but that was the original use case.