4 ms·

Hadoop is amazing for when you need to run huge batch-processing jobs that would otherwise be too taxing to use on a relational database. I've been working a l

by physcab 17y ago

Hadoop is amazing for when you need to run huge batch-processing jobs that would otherwise be too taxing to use on a relational database. I've been working a lot with Hive lately--Facebook's creation that lets you basically type SQL and the Map/Reduce jobs are automatically generated. Hive vastly speeds up development, as you don't need to write custom Map/Reduce jobs to do the tasks you had in mind when working with straight Hadoop. If you're looking to learn more about Hadoop and Hive, Cloudera puts together an awesome starter kit available here: http://www.cloudera.com/distribution http://www.cloudera.com/distribution

jacquesm 17y ago
I failed to find anything related to 'hive' there. One thing completely puzzles me about how hadoop is positioned, on the one hand there is this accent on reliability and scalability, then on the other there is the single meta data node. How does that work in practice, surely there must be some way to have multiple meta-data stores ?
- physcab 17y ago
  oops, wrong link. You want the Hadoop Training for Virtual Machine: http://www.cloudera.com/hadoop-training-virtual-machine http://www.cloudera.com/hadoop-training-virtual-machine It includes Hive pre-installed and Pig as well. It also has a list of tutorial exercises an sample databases for you to practice on. I'm not really a Hadoop expert to be able to address the data store issue. I'm basically one of two researchers at our company that is testing out these new technologies for select tasks (parsing log data, machine learning algorithm implementations, quick statistical munging, etc). In our configuration we have 7 nodes and roughly 3 TB capacity (although we're scaling as our needs increase). For our needs it works quite well. There was an instance where one of our nodes went down while in the middle of a big processing job, and Hadoop kept chugging. Quite amazing.
- voberoi 17y ago
  There is: http://hadoop.apache.org/common/docs/r0.17.2/hdfs_user_guide.html#Secondary+Namenode http://hadoop.apache.org/common/docs/r0.17.2/hdfs_user_guide...
neilc 17y ago
There's no reason you can't use a relational database for "huge batch-processing jobs." You might not be able to use MySQL for such a task, but there are a large collection of DBs that are designed for large-data analytics: Greenplum, Aster Data, Teradata, Vertica, etc. Of course, most of these have a non-zero price tag, but it is certainly possible to use a DB for that sort of workload.
- physcab 17y ago
  Right. I didn't say it wasn't possible to do these with relational databases. Typically you use Hadoop for when you want to take the processing "offline", like in doing data-mining of log data. For our company, our log data is hundreds of millions of rows and if I were to do any significant processing, it would block the ability for anyone else to do any less-intensive tasks. By exporting this data first into HDFS, I can now run Map/Reduce jobs more quickly and not hang up DB connections (we're using MySQL). Another thing to note is that Hadoop becomes incrementally more useful as the size of the files increase--which to my limited DB knowledge, is not the same for relational DB's.
  - neilc 17y ago
    By exporting this data first into HDFS, I can now run Map/Reduce jobs more quickly and not hang up DB connections Right; the typical configuration is to have one database that does transaction processing, and another separate database (a "data warehouse") that collects data from multiple operational DBs and runs analytic queries over it. Hadoop is basically just an alternative analytic query processor in this configuration. Hadoop becomes incrementally more useful as the size of the files increase I think the same would apply to parallel DBs: you are basically talking about just partitioning the data over the storage nodes, which is a common feature in both systems.
- artsrc 17y ago
  If you don't have a primary key and foreign keys you don't really have a relational DB. And if you have a primary key you can't scale perfectly, because you are promising global consistency. A non-zero price tag makes many applications impossible (uneconomic in practice). So that's a good reason to use a non-relational engine for them.

by physcab 17y ago

jacquesm 17y ago

I failed to find anything related to 'hive' there. One thing completely puzzles me about how hadoop is positioned, on the one hand there is this accent on reliability and scalability, then on the other there is the single meta data node. How does that work in practice, surely there must be some way to have multiple meta-data stores ?

physcab 17y ago
oops, wrong link. You want the Hadoop Training for Virtual Machine: http://www.cloudera.com/hadoop-training-virtual-machine http://www.cloudera.com/hadoop-training-virtual-machine It includes Hive pre-installed and Pig as well. It also has a list of tutorial exercises an sample databases for you to practice on. I'm not really a Hadoop expert to be able to address the data store issue. I'm basically one of two researchers at our company that is testing out these new technologies for select tasks (parsing log data, machine learning algorithm implementations, quick statistical munging, etc). In our configuration we have 7 nodes and roughly 3 TB capacity (although we're scaling as our needs increase). For our needs it works quite well. There was an instance where one of our nodes went down while in the middle of a big processing job, and Hadoop kept chugging. Quite amazing.
voberoi 17y ago
There is: http://hadoop.apache.org/common/docs/r0.17.2/hdfs_user_guide.html#Secondary+Namenode http://hadoop.apache.org/common/docs/r0.17.2/hdfs_user_guide...

neilc 17y ago

There's no reason you can't use a relational database for "huge batch-processing jobs." You might not be able to use MySQL for such a task, but there are a large collection of DBs that are designed for large-data analytics: Greenplum, Aster Data, Teradata, Vertica, etc. Of course, most of these have a non-zero price tag, but it is certainly possible to use a DB for that sort of workload.

physcab 17y ago
Right. I didn't say it wasn't possible to do these with relational databases. Typically you use Hadoop for when you want to take the processing "offline", like in doing data-mining of log data. For our company, our log data is hundreds of millions of rows and if I were to do any significant processing, it would block the ability for anyone else to do any less-intensive tasks. By exporting this data first into HDFS, I can now run Map/Reduce jobs more quickly and not hang up DB connections (we're using MySQL). Another thing to note is that Hadoop becomes incrementally more useful as the size of the files increase--which to my limited DB knowledge, is not the same for relational DB's.
- neilc 17y ago
  By exporting this data first into HDFS, I can now run Map/Reduce jobs more quickly and not hang up DB connections Right; the typical configuration is to have one database that does transaction processing, and another separate database (a "data warehouse") that collects data from multiple operational DBs and runs analytic queries over it. Hadoop is basically just an alternative analytic query processor in this configuration. Hadoop becomes incrementally more useful as the size of the files increase I think the same would apply to parallel DBs: you are basically talking about just partitioning the data over the storage nodes, which is a common feature in both systems.
artsrc 17y ago
If you don't have a primary key and foreign keys you don't really have a relational DB. And if you have a primary key you can't scale perfectly, because you are promising global consistency. A non-zero price tag makes many applications impossible (uneconomic in practice). So that's a good reason to use a non-relational engine for them.