4 ms·

There's no reason you can't use a relational database for "huge batch-processing jobs." You might not be able to use MySQL for such a task, but there are a larg

by neilc 17y ago

There's no reason you can't use a relational database for "huge batch-processing jobs." You might not be able to use MySQL for such a task, but there are a large collection of DBs that are designed for large-data analytics: Greenplum, Aster Data, Teradata, Vertica, etc. Of course, most of these have a non-zero price tag, but it is certainly possible to use a DB for that sort of workload.

physcab 17y ago
Right. I didn't say it wasn't possible to do these with relational databases. Typically you use Hadoop for when you want to take the processing "offline", like in doing data-mining of log data. For our company, our log data is hundreds of millions of rows and if I were to do any significant processing, it would block the ability for anyone else to do any less-intensive tasks. By exporting this data first into HDFS, I can now run Map/Reduce jobs more quickly and not hang up DB connections (we're using MySQL). Another thing to note is that Hadoop becomes incrementally more useful as the size of the files increase--which to my limited DB knowledge, is not the same for relational DB's.
- neilc 17y ago
  By exporting this data first into HDFS, I can now run Map/Reduce jobs more quickly and not hang up DB connections Right; the typical configuration is to have one database that does transaction processing, and another separate database (a "data warehouse") that collects data from multiple operational DBs and runs analytic queries over it. Hadoop is basically just an alternative analytic query processor in this configuration. Hadoop becomes incrementally more useful as the size of the files increase I think the same would apply to parallel DBs: you are basically talking about just partitioning the data over the storage nodes, which is a common feature in both systems.
artsrc 17y ago
If you don't have a primary key and foreign keys you don't really have a relational DB. And if you have a primary key you can't scale perfectly, because you are promising global consistency. A non-zero price tag makes many applications impossible (uneconomic in practice). So that's a good reason to use a non-relational engine for them.

by neilc 17y ago

physcab 17y ago

Right. I didn't say it wasn't possible to do these with relational databases. Typically you use Hadoop for when you want to take the processing "offline", like in doing data-mining of log data. For our company, our log data is hundreds of millions of rows and if I were to do any significant processing, it would block the ability for anyone else to do any less-intensive tasks. By exporting this data first into HDFS, I can now run Map/Reduce jobs more quickly and not hang up DB connections (we're using MySQL). Another thing to note is that Hadoop becomes incrementally more useful as the size of the files increase--which to my limited DB knowledge, is not the same for relational DB's.

neilc 17y ago
By exporting this data first into HDFS, I can now run Map/Reduce jobs more quickly and not hang up DB connections Right; the typical configuration is to have one database that does transaction processing, and another separate database (a "data warehouse") that collects data from multiple operational DBs and runs analytic queries over it. Hadoop is basically just an alternative analytic query processor in this configuration. Hadoop becomes incrementally more useful as the size of the files increase I think the same would apply to parallel DBs: you are basically talking about just partitioning the data over the storage nodes, which is a common feature in both systems.

artsrc 17y ago

If you don't have a primary key and foreign keys you don't really have a relational DB. And if you have a primary key you can't scale perfectly, because you are promising global consistency. A non-zero price tag makes many applications impossible (uneconomic in practice). So that's a good reason to use a non-relational engine for them.