3 ms·

Sorry, I didn't mean to offend you in any way! First, I really don't like if somebody by default compares Hadoop with SQL and this is a widespread confusion -

by bitL 13y ago

Sorry, I didn't mean to offend you in any way! First, I really don't like if somebody by default compares Hadoop with SQL and this is a widespread confusion - they are completely different beasts; in fact an extension called Hive gives Hadoop + HTable an SQL-like syntax. However Hadoop is a parallel, batch-processing platform. Hadoop is slow-responding, you can't even talk about latency because a single task takes a lot of time even to setup/execute. It's completely inappropriate for real-time low-latency computations. For those, the in-memory, GPGPU, streams are much better. However, if you have a large dataset whose loading times onto a single machine may be very long - imagine loading 1TB to memory from a network drive if your computer can handle it - you might be better off by partitioning your data across thousands of smaller nodes (e.g. ARM microservers) and perform computations on each of them independently. This way you don't need to transfer a lot of data, each smaller local dataset is loaded very fast (= you preserver locality), you find balance between being CPU-bound and IO-bound and likely finish your computations much faster than on a single computer with huge memory but limited bus. Hadoop's tragedy is that it is now an established platform that is supported by large companies which are mostly driven by technologically clueless people, trying to put it everywhere as it is "in vogue" to do so. Google abandoned MapReduce model a few years ago but industry didn't seem to notice. Think about your case for chemical structures - is there any part of your algorithm that needs to be computed only occasionally but it's a lot of data to process? Why not offload it to Hadoop for preparing digest from these data which you can use in your real-time algorithm? That's actually pretty common usage scenario - Hadoop handles the rough mining, extracting the precious stuff from crude data, assembling it to a form that is refined and can be used by other parts of system that have completely different requirements, for example low-latency interactions.

dalke 13y ago
> "I really don't like if somebody by default compares Hadoop with SQL" The first comparison was to Scala, and then to SQL. The main comparison was to SQL and a Python script. While it does talk more about SQL than other solutions, I don't see that as a default comparison. > "For those, the in-memory, GPGPU, streams are much better" I'll go off on a bit of philosophical tangent here. Yes, GPGPUs can be more effective for the task I'm doing, since I'm actually memory bound. However, GPGPUs require dedicated hardware, while the code I work is effective even on laptops with limited GPUs, including web servers. Ideologically, I prefer to enable single-person developers, and scientists who do not have much training in hardware and network administration. For that situataion, GPGPUs are not "much better", because so long as the performance is fast enough, it doesn't need to be faster. And 100 ms is "fast enough." > "This way you don't need to transfer a lot of data" True. But to point out, I used the traditional approach of developing a file format which can be memory-mapped directly to my internal data structures, and a search algorithm which doesn't need to search the entire data set before loading it. These optimizations aren't synergistic with how I understand how Hadoop works. > Hadoop's tragedy ... mostly driven by technologically clueless people The author of the essay agrees with you. Here's the P.P.S.: "I don’t intend to hate on Hadoop. I use Hadoop regularly for jobs I probably couldn’t easily handle with other tools. ... Hadoop is a fine tool, it makes certain tradeoffs to target certain specific use cases. The only point I’m pushing here is to think carefully rather than just running Hadoop on The Cloud in order to handle your 500mb of Big Data at an Enterprise Scale." This is why I don't think you actually disagree with the author. > Think about your case for chemical structures - is there any part of your algorithm that needs to be computed only occasionally but it's a lot of data to process? Certainly, but there are two other important facets to that. 1) updates occur weekly, a full rebuild on a single core only takes 12 hours, and those 12 hours aren't critical, and 2) the deltas are relatively small and data from previous builds can be reused, so incremental updates should take about 30 minutes - I'll be working on that code in the next couple of weeks. It's easier to have a cron job trigger a command-line program every week than to set up a Hadoop server.

by bitL 13y ago

dalke 13y ago

> "I really don't like if somebody by default compares Hadoop with SQL" The first comparison was to Scala, and then to SQL. The main comparison was to SQL and a Python script. While it does talk more about SQL than other solutions, I don't see that as a default comparison. > "For those, the in-memory, GPGPU, streams are much better" I'll go off on a bit of philosophical tangent here. Yes, GPGPUs can be more effective for the task I'm doing, since I'm actually memory bound. However, GPGPUs require dedicated hardware, while the code I work is effective even on laptops with limited GPUs, including web servers. Ideologically, I prefer to enable single-person developers, and scientists who do not have much training in hardware and network administration. For that situataion, GPGPUs are not "much better", because so long as the performance is fast enough, it doesn't need to be faster. And 100 ms is "fast enough." > "This way you don't need to transfer a lot of data" True. But to point out, I used the traditional approach of developing a file format which can be memory-mapped directly to my internal data structures, and a search algorithm which doesn't need to search the entire data set before loading it. These optimizations aren't synergistic with how I understand how Hadoop works. > Hadoop's tragedy ... mostly driven by technologically clueless people The author of the essay agrees with you. Here's the P.P.S.: "I don’t intend to hate on Hadoop. I use Hadoop regularly for jobs I probably couldn’t easily handle with other tools. ... Hadoop is a fine tool, it makes certain tradeoffs to target certain specific use cases. The only point I’m pushing here is to think carefully rather than just running Hadoop on The Cloud in order to handle your 500mb of Big Data at an Enterprise Scale." This is why I don't think you actually disagree with the author. > Think about your case for chemical structures - is there any part of your algorithm that needs to be computed only occasionally but it's a lot of data to process? Certainly, but there are two other important facets to that. 1) updates occur weekly, a full rebuild on a single core only takes 12 hours, and those 12 hours aren't critical, and 2) the deltas are relatively small and data from previous builds can be reused, so incremental updates should take about 30 minutes - I'll be working on that code in the next couple of weeks. It's easier to have a cron job trigger a command-line program every week than to set up a Hadoop server.