3 ms·

I understand and agree with the author's main point that many companies that use big data do not need to use these technologies. I do not agree that the tools

I understand and agree with the author's main point that many companies that use big data do not need to use these technologies. I do not agree that the tools are inferior to Sql. Hive is really close to sql and Pig is extremely powerful. I would take a look at a few of the recent updates to these tools before declaring them inferior to Sql.

ssalbiz 13y ago
Disclaimer: I've worked with and submitted a few odd patches to Hive. I have not worked with Pig directly. I think you ought to consider what you mean by inferior here carefully. If you mean 'Hive QL can capture many of the common semantics of SQL' then sure. If you mean just about anything else, you're wrong. The performance and reliability considerations of Hive/Hadoop are vastly different and very easily inferior to a mysql or postgres setup for small to mid-size datasets. (That doesn't even get into ease-of-usage. Anyone who's ever dealt with Hive's dreaded 'error: return code: -9' can attest to how maddening Hive can be to use).
acdha 13y ago
There's a key nuance missing: SQL is mature, while even core Hadoop is struggling to get there. You can simply install Postgres, MySQL, etc. in a couple minutes and start working on your data (i.e. the actual work) and not spend hours dealing with … mixed quality … documentation, extensive configuration on multiple nodes, and writing code to provide what are built-in features in most databases. For anything not set in stone with massive data volumes, that overhead adds up quickly. The other cost is interactivity: Hive takes a LONG time to return results compared to a SQL database even if you don't have massive data. If an analyst is working on a query interactively this is a significant impediment, particularly given the gap in ease of monitoring and performance optimization advice. Again, if you have enough data Hadoop might still be worth it but, particularly in the post-SSD big memory era, the time-to-correct result gap is massive.

ssalbiz 13y ago

Disclaimer: I've worked with and submitted a few odd patches to Hive. I have not worked with Pig directly. I think you ought to consider what you mean by inferior here carefully. If you mean 'Hive QL can capture many of the common semantics of SQL' then sure. If you mean just about anything else, you're wrong. The performance and reliability considerations of Hive/Hadoop are vastly different and very easily inferior to a mysql or postgres setup for small to mid-size datasets. (That doesn't even get into ease-of-usage. Anyone who's ever dealt with Hive's dreaded 'error: return code: -9' can attest to how maddening Hive can be to use).

acdha 13y ago

There's a key nuance missing: SQL is mature, while even core Hadoop is struggling to get there. You can simply install Postgres, MySQL, etc. in a couple minutes and start working on your data (i.e. the actual work) and not spend hours dealing with … mixed quality … documentation, extensive configuration on multiple nodes, and writing code to provide what are built-in features in most databases. For anything not set in stone with massive data volumes, that overhead adds up quickly. The other cost is interactivity: Hive takes a LONG time to return results compared to a SQL database even if you don't have massive data. If an analyst is working on a query interactively this is a significant impediment, particularly given the gap in ease of monitoring and performance optimization advice. Again, if you have enough data Hadoop might still be worth it but, particularly in the post-SSD big memory era, the time-to-correct result gap is massive.