4 ms·
Right. I didn't say it wasn't possible to do these with relational databases. Typically you use Hadoop for when you want to take the processing "offline", lik
by physcab 17y ago
Right. I didn't say it wasn't possible to do these with relational databases. Typically you use Hadoop for when you want to take the processing "offline", like in doing data-mining of log data. For our company, our log data is hundreds of millions of rows and if I were to do any significant processing, it would block the ability for anyone else to do any less-intensive tasks. By exporting this data first into HDFS, I can now run Map/Reduce jobs more quickly and not hang up DB connections (we're using MySQL).
Another thing to note is that Hadoop becomes incrementally more useful as the size of the files increase--which to my limited DB knowledge, is not the same for relational DB's.
- neilc 17y agoBy exporting this data first into HDFS, I can now run Map/Reduce jobs more quickly and not hang up DB connections Right; the typical configuration is to have one database that does transaction processing, and another separate database (a "data warehouse") that collects data from multiple operational DBs and runs analytic queries over it. Hadoop is basically just an alternative analytic query processor in this configuration. Hadoop becomes incrementally more useful as the size of the files increase I think the same would apply to parallel DBs: you are basically talking about just partitioning the data over the storage nodes, which is a common feature in both systems.