5 ms·

You are absolutely correct. MapReduce is chiefly a framework for performing distributed computation. RDBMSes, colloquially referred to as "SQL databases", are

by pfedor 18y ago

You are absolutely correct. MapReduce is chiefly a framework for performing distributed computation. RDBMSes, colloquially referred to as "SQL databases", are for data storage and (parallel, transactional, ...) access. If you want to compare SQL databases with some Google technology, probably the closest thing would be http://labs.google.com/papers/bigtable.html http://labs.google.com/papers/bigtable.html Bigtable. MapReduce doesn't even come close to being anything comparable.

neilc 18y ago
MapReduce and parallel SQL implementations are perfectly comparable, as the paper itself makes clear: both are for doing large-scale data analysis. You can use RDBMSs for other things, but it is perfectly feasible to compare shared nothing massively parallel SQL implementations with MapReduce, because they are often used for similar purposes.
- pfedor 18y ago
  You can write a paper comparing Perl with MySQL, and if you put enough effort into it you may even conclude that MySQL has better performance, but this doesn't make the comparison any more meaningful.
  - neilc 18y ago
    How is the comparison not meaningful? MapReduce and parallel SQL implementations (e.g. Greenplum, Teradata) are competitors, and are typically used for precisely the same task (large-scale data analysis). They obviously differ in some ways (which is what makes a comparison interesting), but your suggestion that "MapReduce doesn't even come close to being anything comparable" is just uninformed.
    - pfedor 18y ago
      I'm surprised by the direction this conversation is taking. I assumed the confusion came from a misunderstanding of what MapReduce was, since it's a much less known technology. My comment was intended to clarify this confusion. I didn't expect a disagreement about what a relational database is. I've never heard of Greenplum and Teradata. I had however used a number of RDBMSes for five years at three different jobs. I read a book and took a class on relational databases. Persistent storage and concurrent access were central features of any database I've ever used or learned about. The databases the authors of the article use seem to be of the same kind. For example, they first load the data into a database and then use SQL to query it. There are no equivalent concepts in the MapReduce framework. It doesn't make any sense, for example, to talk about loading data into MapReduce. A map reduction is typically a process that takes input from some place in some form (could even be a relational database), performs some computation on it, and writes the output in another (possibly identical) form. I don't see how you can say the two are competitors, any more than Perl is a competitor to MySQL. I actually know quite well what MapReduce is typically used for, and I strongly doubt relational databases are used for precisely the same (or even somewhat similar) tasks.
      - neilc 18y ago
        I've never heard of Greenplum and Teradata. They are parallel databases, which is what the article is comparing MapReduce against. The basic idea is that you partition the data over a set of machines, and similarly divide and parallelize query processing. MR is basically a specialized parallel DB that executes a single query of the form: SELECT map(...), reduce_agg(...) FROM ... GROUP BY map(...) Again, MR's architecture differs in various ways from the design of a typical parallel DB, but they are certainly comparable. It doesn't make any sense, for example, to talk about loading data into MapReduce MR input data is typically stored on a distributed filesystem (e.g. HDFS, GFS). Loading the data onto that filesystem is analogous to loading the data into a parallel DB; in both cases, the input data is partitioned over a set of nodes. For good performance, you want a copy of the data to reside close to where the subsequent computation is going to take place. But certainly MR and DBs differ in this respect: a DB typically assumes it "owns" the data, whereas MR is more amenable to processing data stored in an arbitrary external format. This difference is not fundamental, though. I strongly doubt relational databases are used for precisely the same (or even somewhat similar) tasks. You would be mistaken, then. If you want to do analysis on large volumes of data using a cluster of machines, both parallel DBs and MR are viable choices. The whole point of the paper is to take a set of queries, execute them using both Hadoop and parallel DBs, and compare the resulting performance and user experience.

by pfedor 18y ago

neilc 18y ago

MapReduce and parallel SQL implementations are perfectly comparable, as the paper itself makes clear: both are for doing large-scale data analysis. You can use RDBMSs for other things, but it is perfectly feasible to compare shared nothing massively parallel SQL implementations with MapReduce, because they are often used for similar purposes.

pfedor 18y ago
You can write a paper comparing Perl with MySQL, and if you put enough effort into it you may even conclude that MySQL has better performance, but this doesn't make the comparison any more meaningful.
- neilc 18y ago
  How is the comparison not meaningful? MapReduce and parallel SQL implementations (e.g. Greenplum, Teradata) are competitors, and are typically used for precisely the same task (large-scale data analysis). They obviously differ in some ways (which is what makes a comparison interesting), but your suggestion that "MapReduce doesn't even come close to being anything comparable" is just uninformed.
  - pfedor 18y ago
    I'm surprised by the direction this conversation is taking. I assumed the confusion came from a misunderstanding of what MapReduce was, since it's a much less known technology. My comment was intended to clarify this confusion. I didn't expect a disagreement about what a relational database is. I've never heard of Greenplum and Teradata. I had however used a number of RDBMSes for five years at three different jobs. I read a book and took a class on relational databases. Persistent storage and concurrent access were central features of any database I've ever used or learned about. The databases the authors of the article use seem to be of the same kind. For example, they first load the data into a database and then use SQL to query it. There are no equivalent concepts in the MapReduce framework. It doesn't make any sense, for example, to talk about loading data into MapReduce. A map reduction is typically a process that takes input from some place in some form (could even be a relational database), performs some computation on it, and writes the output in another (possibly identical) form. I don't see how you can say the two are competitors, any more than Perl is a competitor to MySQL. I actually know quite well what MapReduce is typically used for, and I strongly doubt relational databases are used for precisely the same (or even somewhat similar) tasks.
    - neilc 18y ago
      I've never heard of Greenplum and Teradata. They are parallel databases, which is what the article is comparing MapReduce against. The basic idea is that you partition the data over a set of machines, and similarly divide and parallelize query processing. MR is basically a specialized parallel DB that executes a single query of the form: SELECT map(...), reduce_agg(...) FROM ... GROUP BY map(...) Again, MR's architecture differs in various ways from the design of a typical parallel DB, but they are certainly comparable. It doesn't make any sense, for example, to talk about loading data into MapReduce MR input data is typically stored on a distributed filesystem (e.g. HDFS, GFS). Loading the data onto that filesystem is analogous to loading the data into a parallel DB; in both cases, the input data is partitioned over a set of nodes. For good performance, you want a copy of the data to reside close to where the subsequent computation is going to take place. But certainly MR and DBs differ in this respect: a DB typically assumes it "owns" the data, whereas MR is more amenable to processing data stored in an arbitrary external format. This difference is not fundamental, though. I strongly doubt relational databases are used for precisely the same (or even somewhat similar) tasks. You would be mistaken, then. If you want to do analysis on large volumes of data using a cluster of machines, both parallel DBs and MR are viable choices. The whole point of the paper is to take a set of queries, execute them using both Hadoop and parallel DBs, and compare the resulting performance and user experience.