4 ms·
They are different beasts really. It'd be hard to implement something complex, like the PageRank algorithm, in SQL. On the other hand, MapReduce is not designed
by andr 18y ago
They are different beasts really. It'd be hard to implement something complex, like the PageRank algorithm, in SQL. On the other hand, MapReduce is not designed for tasks such as "SELECT * FROM table WHERE field > 5", which is one of the benchmarks the paper uses.
Also, parallel DBMSs don't handle failure well. Ask any of the MapReduce developers at Google and they'll tell you failure handling is the hardest part of the problem.
With all that said, I'd be very interested to see a comparison between Vertica, DBMS-X and MySQL, running in sharded or MySQL Cluster mode.
PS The paper is coauthored by Microsoft.
- gaius 18y agoAlso, parallel DBMSs don't handle failure well. It depends on the type of failure. You can simply skip offline partitions, for example. PS The paper is coauthored by Microsoft. Microsoft were doing shared-nothing clustering on SQL Server LONG before MySQL types had even coined the word sharding to mean the same thing...
- andr 18y agoMicrosoft were doing shared-nothing clustering on SQL Server LONG before MySQL types had even coined the word sharding to mean the same thing... Microsoft are heavily invested in SQL Server, so they are biased against MapReduce.
- gaius 18y agoMicrosoft are heavily invested in SQL Server, so they are biased against MapReduce. Microsoft is a huge company and parts of it a biased in favour of things like this (e.g. the bits working on F#)
- eob 18y agoThe paper was co-authored by David DeWitt, not Microsoft. DeWitt is a well-known database researcher from Wisconsin. And Microsoft, despite their awful history of business practices, pours tons of funding into really good quality research (via MSR). I'm not saying people aren't biased by their associations -- certainly, the traditional DBMS community feels like they have to defend their research & products against new techniques like MapReduce.. but if you're going to criticize the paper, criticize it because you found flaws in its contents, not because of the organization listed beneath an author.
- warfangle 18y agoAre you familiar with any projects currently using Vertica, and how well it's worked for them?
- briansmith 18y agoA single-statement SQL PageRank implementation sounds feasible to me. Most commercial SQL implementations have had hierarchal, ranking, and windowing functions for a while now. They all have XML (maybe HTML?) document parsers in them as well. I think moderately complex hierarchical rankings can be built from those primitives, even as a single SQL statement. The performance of that code and if/how they are parallelized by the system is really a quality-of-implementation issue. If Oracle, Microsoft, or IBM thinks they can make money by implementing automatically parallelized hierarchical ranking and windowing functions, they will do so; maybe some of them have already done so. I share Oracle's philosophy about how abstract SQL should be. Oracle says you should be able to write the same SQL against a small single-instance database that you would write against a giant multi-instance database; the database is responsible for optimizing the performance, not the developer. There are a lot of cases where the theory doesn't match practice with Oracle, but every new release gets the two closer. I wouldn't be surprised to see Oracle surpass general-purpose MapReduce-based approaches in terms of performance soon. And, I am expecting Oracle to release a tool to efficiently run MapReduce-style programs against Oracle (converting them to SQL) any time now.