3 ms·

I worked on Hadoop and HBase extensively from 2011 - 2013, working on engines processing 30 billion raw data points a month and storing a subset of those, and t

I worked on Hadoop and HBase extensively from 2011 - 2013, working on engines processing 30 billion raw data points a month and storing a subset of those, and then we migrated to other Big Data technologies. Just wanted to add my thoughts here. Hadoop (and its general ecosystem, which includes HBase), is a fairly good idea. Its core ideas - map/reduce on Hadoop, and a large distributed key/value store for HBase - are actually pretty solid. And for many years, there were simply no alternatives to Hadoop. Think of the years 2008 to 2012/13. If you had to process terabytes or petabytes of data, what were your solutions? No wonder Yahoo and Facebook (and others) put in so much effort into their Hadoop solutions. But, IMHO, there were several issues with Hadoop and their ilk. 1. The core infrastructure wasn't stable enough. Hadoop / HBase were supposed to be distributed systems, and they worked well, but small failures could bring down your entire cluster. Given that Hadoop and HBase were being used in mission-critical systems in the cloud, and given the amount of DevOps or sys-admin work that went into maintaining these, I'm not surprised people eventually migrated to distributed systems that were easier to maintain and run. 2. There are now plenty of "hosted on the cloud" solutions such as Amazon DynamoDB or similar cloud solutions. When your company depends on 99.99% or similar SLAs, you don't want to have downtime on your database systems and spend time debugging complicated core dumps on your Hadoop or HBase clusters when you can just store it "on the cloud" and be done with it. Sure, there's a higher price point, but those are the trade-offs you live with. 3. If you want to be in-house, there are plenty of alternatives out there as well today. Apache Spark for processing, Kafka for a messaging bus / streaming data, ElasticSearch for large scale storage, with multiple indices. Many of them are much more robust than Apache Hadoop / HBase, and I'm not surprised they've gotten more traction recently. Ultimately, I think Hadoop / HBase are just showing their age. They were fantastic for the first wave of Big Data technologies, and you had little alternative if you were building large-scale systems circa 2008 to 2013, but now, you just have a plethora of choices from various vendors.

toast0 10y ago
Another factor is that memory density has increased a lot since 2008; you can easily get 1TB of ram in a dual socket system, so for memory space limited jobs, you can run on fewer machines, and maybe just one, if your needs haven't grown as fast as technology has.
sixdimensional 10y ago
Just curious, what did you migrate to? I think the comment "first wave of Big Data technologies" is interesting. I agree with it, but also, the term "big data" is relevant to a point in time, I think. I mean, VLDB was the term before "big data" - I think when the term "big data" became popular was when technology such as Hadoop became more mainstream and available instead of just being used in special cases. I was working with terabytes of data in SQL Server prior to 2008 (around 2004/2005 - which was doable but pushing some limits) and then got into Hadoop etc. afterwards. It was interesting to reach the limits of a tech we were using and be there for the transition to Hadoop etc. There were expensive commercial alternatives - MPP databases like Teradata, Netezza or Vertica - or things like scientific computing platforms (which were very limited to scientific computing scenarios). But nothing was as.. mainstream and available as Hadoop became. And yet, Hadoop never really completely beat the traditional MPP vendors until you started getting engines like Impala, Presto, Spark etc. that made SQL on top of Hadoop easier. And that took quite a while.. in fact I'd say making SQL on top of Hadoop work well (among many other things) is probably still on-going in a way. We are much, much further along with that now than we were. On the other hand, it was interesting to see the new opportunities develop for working with data other than just SQL - straight MapReduce, machine learning, graph processing, DAG, etc. Will be interesting to see where we go from here. I still think there is a lot going on.
- samspenc 10y ago
  We were running heavily on Amazon AWS, and I changed jobs shortly before the migration, but I think they moved to a variety of Amazon solutions (Kinesis, RedShift, etc). Not sure what they are running on now. To be honest, Hadoop on AWS through Elastic MapReduce (EMR) actually served us really well. The cluster downtime we had in our early days were all "human error" (configuration and Map/Reduce programming mistakes). Once we fixed those, EMR served us like a champ. The bigger issues we had were with HBase actually, particularly during its region splitting. It was great for storing massive amounts of data that you could store and query fast (with the right key design), but we weren't able find people skilled enough in HBase ops to make us comfortable with continuing to use it. I have seen Teradata, Netezza and Vertica being used heavily at that time (circa 2010-2013) but never worked on them directly. Like you said, they are extremely expensive, and I can imagine why some of the larger installations were trying to move to alternatives. End of the day, I don't have any perfect solutions to recommend to Hadoop or HBase. I think there are a variety of cloud-based solutions from Amazon, Microsoft and Google; as well as hybrid open-source solutions that do the job, but to each his own!
neeleshs 10y ago
We have a system based on hbase with similar volume, with my elasticsearch experience, I cannot really see it replace hbase at these volumes. Also, Kafka is not a replacement, but complementary. The only other mature onprem store I can think of is Cassandra. There are other kids on the block, aerospike, cockroach db etc, with different guarantee semantics. I do agree hbase is a monster to manage. When it works though, there aren't many systems which can mach it's insert/scan capabilities T scale.
- samspenc 10y ago
  Agreed, I've seen reports on ElasticSearch's website that their largest customers are running petabyte-scale data on ElasticSearch clusters, but my rule of thumb has been: HBase for petabyte-scale data, ElasticSearch and others for terabyte-scale data. As for Kafka, I was surprised to see the article lump this together with Hadoop. I've used Kafka exclusively as a messaging bus or to stream large amounts of data. It doesn't exist as a replacement to Hadoop or Big Data technologies, but as a perfect complement, because it doesn't process data, just streams / moves it efficiently.

toast0 10y ago

Another factor is that memory density has increased a lot since 2008; you can easily get 1TB of ram in a dual socket system, so for memory space limited jobs, you can run on fewer machines, and maybe just one, if your needs haven't grown as fast as technology has.

sixdimensional 10y ago

Just curious, what did you migrate to? I think the comment "first wave of Big Data technologies" is interesting. I agree with it, but also, the term "big data" is relevant to a point in time, I think. I mean, VLDB was the term before "big data" - I think when the term "big data" became popular was when technology such as Hadoop became more mainstream and available instead of just being used in special cases. I was working with terabytes of data in SQL Server prior to 2008 (around 2004/2005 - which was doable but pushing some limits) and then got into Hadoop etc. afterwards. It was interesting to reach the limits of a tech we were using and be there for the transition to Hadoop etc. There were expensive commercial alternatives - MPP databases like Teradata, Netezza or Vertica - or things like scientific computing platforms (which were very limited to scientific computing scenarios). But nothing was as.. mainstream and available as Hadoop became. And yet, Hadoop never really completely beat the traditional MPP vendors until you started getting engines like Impala, Presto, Spark etc. that made SQL on top of Hadoop easier. And that took quite a while.. in fact I'd say making SQL on top of Hadoop work well (among many other things) is probably still on-going in a way. We are much, much further along with that now than we were. On the other hand, it was interesting to see the new opportunities develop for working with data other than just SQL - straight MapReduce, machine learning, graph processing, DAG, etc. Will be interesting to see where we go from here. I still think there is a lot going on.

samspenc 10y ago
We were running heavily on Amazon AWS, and I changed jobs shortly before the migration, but I think they moved to a variety of Amazon solutions (Kinesis, RedShift, etc). Not sure what they are running on now. To be honest, Hadoop on AWS through Elastic MapReduce (EMR) actually served us really well. The cluster downtime we had in our early days were all "human error" (configuration and Map/Reduce programming mistakes). Once we fixed those, EMR served us like a champ. The bigger issues we had were with HBase actually, particularly during its region splitting. It was great for storing massive amounts of data that you could store and query fast (with the right key design), but we weren't able find people skilled enough in HBase ops to make us comfortable with continuing to use it. I have seen Teradata, Netezza and Vertica being used heavily at that time (circa 2010-2013) but never worked on them directly. Like you said, they are extremely expensive, and I can imagine why some of the larger installations were trying to move to alternatives. End of the day, I don't have any perfect solutions to recommend to Hadoop or HBase. I think there are a variety of cloud-based solutions from Amazon, Microsoft and Google; as well as hybrid open-source solutions that do the job, but to each his own!

neeleshs 10y ago

We have a system based on hbase with similar volume, with my elasticsearch experience, I cannot really see it replace hbase at these volumes. Also, Kafka is not a replacement, but complementary. The only other mature onprem store I can think of is Cassandra. There are other kids on the block, aerospike, cockroach db etc, with different guarantee semantics. I do agree hbase is a monster to manage. When it works though, there aren't many systems which can mach it's insert/scan capabilities T scale.

samspenc 10y ago
Agreed, I've seen reports on ElasticSearch's website that their largest customers are running petabyte-scale data on ElasticSearch clusters, but my rule of thumb has been: HBase for petabyte-scale data, ElasticSearch and others for terabyte-scale data. As for Kafka, I was surprised to see the article lump this together with Hadoop. I've used Kafka exclusively as a messaging bus or to stream large amounts of data. It doesn't exist as a replacement to Hadoop or Big Data technologies, but as a perfect complement, because it doesn't process data, just streams / moves it efficiently.