3 ms·
The other point that's commonly missed is that Hadoop is really only useful when both inputs and outputs are too large to fit on one machine. If you have a big
by nostrademons 9y ago
The other point that's commonly missed is that Hadoop is really only useful when both inputs and outputs are too large to fit on one machine. If you have a big-data input but a small-data output (which is very common in a lot of exploratory problems), you can get away with a simple work-queue setup that sends results into a shared RDBMS or filesystem.
At the beginning of my current project, I had a job that involved 35T of input, but the vast majority of records would be ignored, and then for each successful one, only a few hundred bytes of output would be generated. Rather than Hadoop, I setup a simple system where a number of worker processes would query Postgres for the next available shard, mark it as in-progress, and then stream it from S3 and process it. When they finished, they'd write a CSV file back to S3. The reduce phase was just 'cat'.
The resulting system took a few hours to build (few days, including the actual algorithms run), and it was much more debuggable than Hadoop would be. You could inspect exactly where the job was, what shards had errored out, and which were currently running on machines, and download & view intermediate results before the whole computation finished. You could run the workers locally on a MBP if you needed to debug a shard, with no setup needed.
When I was at Google, we had a saying that "The only interesting part of MapReduce is the phase that's not in the name: the Shuffle". [That's the phase where the outputs of the Map are sorted, written to the filesystem and eventually network, and delivered to the appropriate Reduce shard.] If you don't need a shuffle phase - either because you have no reducer, your reduce input is small enough to fit on one machine, or your reduce input comes infrequently enough that a single microservice can keep up with all the map tasks - then you don't need a MapReduce-like framework.
- dsacco 9y agoYes, very strong agree. I currently use HDF5 on my home machine that sucks in a fairly large amount of data (it's in the tens of TB, and adds 20GB daily). Before I set up HDF5 my database experience was mostly limited to Postgres (although more recently MySQL). I explored a variety of options including Hadoop and Cassandra but I just didn't really want to have more than one node for this exact reason, and I couldn't see a compelling advantage to either without that sort of workflow. Were I not working specifically with timeseries, I probably would have thrown it into postgres. There's a particular bias I come across where people who genuinely have big data want to set it up in ways that is not necessarily performant because Hadoop is basically the most recognizable tool. If the data processing I'm doing generated a lot of output data I might consider a different flow, but there just isn't much of a reason: most of the data is inert for long periods of time, the output insights are fairly small and the actual processing has to occur very quickly and with low latency.
- lmm 9y ago> it was much more debuggable than Hadoop would be. You could inspect exactly where the job was, what shards had errored out, and which were currently running on machines, and download & view intermediate results before the whole computation finished. You could run the workers locally on a MBP if you needed to debug a shard, with no setup needed. To the extent that's true it's an indictment of Hadoop's implementation. Doing all those things in Hadoop ought to be trivial; maybe there are a few tools you'd have to make a one-time effort to learn, but reusing them ought to save you effort over making a custom system every time.
- acdha 9y agoThat was certainly the case in the past. We backed away from it ~5 years ago after wasting too much time investigating cases where a daemon had leaked memory, hit the heap limit, and failed in a way which caused work to halt but produced no visible error state other than a stack trace in one of the many log files. Local testing wasn't viable since you needed something like 16-20GB to run stably, and back then we didn't have individual dev systems with enough RAM to run that kind of overhead and still have enough left for the actual work.