4 ms·
Keep in mind that the article itself admits that the loading the data into an RDBMS took much longer than loading into onto a Hadoop cluster. That's what the m
by strlen 18y ago
Keep in mind that the article itself admits that the loading the data into an RDBMS took much longer than loading into onto a Hadoop cluster.
That's what the main crux is: if your data isn't relational to start with (e.g. messages in a message queue, being written to files in HDFS), you're better of with Map/Reduce. If your data is relational to start with, you're better off doing traditional OLAP on an RDBMs cluster.
When I worked for a start-up ad network, we couldn't afford the latency/contention implications of writing to a database each time we served an ad. Initially, we moved the model to "log impressions to text files, rsync text files over, run a cron job to enter them into an RDBMS". Problem is, that processing the data (on a single node) was taking so long the data would no longer by meaningful by the time it would be in RDMBS (and OLAP queries would finish running).
I thought of "how can I ensure that a specific machine only processes a specific log file, or a specific section of a log file" -- but then I realized that's what the map part of map/reduce is. In the end I've setup a Hadoop cluster, which made both "soft real time" (hourly/bi-hourly) data processing and long term queries (running on multiple month, multi-terrabyte datasets) much faster and easier.
Theoretically, yes: RDBMS is the most efficient way to run complex analytics queries. Practically, we have to handle such issues as asynchronicity, contention, ease of scalability and getting the data into a relational format (which itself is something that could be done using Hadoop: one of the tasks we used it for was transforms all sorts of text data -- crawled webpages, log files -- into CSV data that could be loaded onto the OLAP/OLTP clusters with "LOAD DATA INFILE").
- neilc 18y agoif your data isn't relational to start with (e.g. messages in a message queue, being written to files in HDFS), you're better of with Map/Reduce If you query the data once and then throw it away, then certainly load performance is a critical issue. If you load the data once and then query it many times, load performance is far less important -- trading a longer load time for much better query performance would be a good idea. So it depends on the workload as much as the format the data happens to start in.
- strlen 18y agoI think Hadoop and M/R seem to be best for: "Take the data, transform it into another form ONCE, query the transformed using a low latency scheme many times". (E.g. building a web index or in my case, building per ad/per user/per publisher key/value pairs for ad targeting).