2 ms·
Preface: "Big Data" is a stupid label, and I wish it would die. You have no idea how much time I spend explaining to non-technical people that comparing Hadoop
by alanctgardner2 13y ago
Preface: "Big Data" is a stupid label, and I wish it would die. You have no idea how much time I spend explaining to non-technical people that comparing Hadoop and Riak is like comparing a tractor to a helicopter.
You can definitely put lots of data in a RDBMS, and you can get it back very quickly. I'm not advocating for Hadoop for problems where a database - be it sharded, vertically scaled, whatever - will do; I know you can scale relational to petabytes of data.
Hadoop isn't supposed to be a tool for having a lot of data, it's a tool for doing things with a lot of data. I've done lots of weird computation on Hadoop: OCR of time-series images is a good example. This is general purpose, parallel computation which takes advantage of Hadoop's scheduling infrastructure and the notion of data locality - some of the data lives on every compute node, and can be accessed with very low latency.
To put it another way, databases will get faster and more scalable, but there's still a need for ETL when moving between different data models. Some people use Hadoop just for ETL, particularly when ingesting semi-structured data, because it's good at computation, but only OK at storing the finished, structured data.
I'm going to try not to be offended by your closing line; I was trying to point out a particular dimension of considering whether an application is suited to Hadoop - the computation component. This is often ignored by people who treat Hadoop like yet another database, when that's really a complete mischaracterization. I certainly don't advocate 'relying on any mantra', but instead considering all solutions and selecting the most appropriate tool.