4 ms·

Worked at Hortonworks and post merger Cloudera. Was interesting to see how market demand changed over the years and how the company worked on re-inventing thems

by wardb 5y ago

Worked at Hortonworks and post merger Cloudera. Was interesting to see how market demand changed over the years and how the company worked on re-inventing themselves. Databricks and Snowflake seemed to understand SaaS earlier and better though. Still have some great friends working at Cloudera, and hope this will indeed accelerate a great next phase.

swyx 5y ago
for the uninitiated, do you mind diffing what Hortonworks used to do and what post merger Cloudera is now focused on?
- threeseed 5y ago
  Before the cloud took off Hortonworks and Cloudera owned the Big Data market. They both offered a Hadoop distribution but had different strengths e.g. Hortonworks had fine grained access control, Cloudera had a better SQL product with Impala. Then AWS came along and built their own which was significantly cheaper and more flexible as you could easily scale your cluster up/down. And so companies moved to it when they over time began to move to the cloud. The Hortonworks/Cloudera response to this threat was to put away their differences and merge together. Over time Big Data has evolved from being Hadoop centric to being much more ML/AI focused i.e. not just manipulating and querying the data but doing something interesting with it. And AWS, Azure, GCP have really jumped in with a whole suite of products that are tightly integrated with the rest of their cloud offerings. And it's a large part of what differentiates their offerings so they compete very hard. So Cloudera has no choice but to do things that cloud providers won't or can't do: (1) focus on non-cloud or multi-cloud and (2) offer a much more integrated and cohesive solution. But having spent 10+ years in this space and deployed many Hadoop clusters I can tell you that Cloudera is going to struggle. Companies that I never thought would move to the cloud e.g. banks are figuring out the security and regulatory challenges and eagerly moving across. And so it's going to be a Cloudera versus Amazon/Google/Microsoft which is an impossible fight.
  - swyx 5y ago
    thank you! i have no history in this space so can't ask followups except to observe that the tendency of ML/AI to reward the "big gets bigger" phenomenon is exemplified here. I don't feel too great about that but also don't have ideas for a better system.
  - wardb 5y ago
    Competing with Amazon/Google/Microsoft on their own cloud is...ehm...good luck with that indeed. I believe they should have partnered with them early on (real partnership, like a premier offering, not the rubber stamp / marketplace partnership).
    - manigandham 5y ago
      It can work, as that's exactly what Snowflake has done and it's one of the fastest-growing SaaS companies today. A good product is more valuable than a partnership.
      - 8ytecoder 5y ago
        Databricks as well. Azure sells both their own version and “Azure Databricks”
      - wardb 5y ago
        Good products, but don't discount their go-to-market strategy. IIRC MS/Azure is an early Databricks investor and their sales folks were heavily incentivised to sell it. They also pushed Snowflake in the early days until they had a competing product and their relationship status was upgraded to 'it's complicated'.
  - bostonsre 5y ago
    AWS EMR is still pretty pricey compared to free ambari/cloudera running on ec2. Although, there is a lot of time and effort that needed to be put into automation that uses those ambari/cloudera hadoop management layers. After they merged, they got really aggressive and made moves that effectively killed each of the free versions. They definitely put another nail in the coffin of hadoop. Spark on kubernetes is pretty gorgeous and has been a successful route out of pricey hadoop infrastructure for my company.
- macksd 5y ago
  Worked at Cloudera pre- and post-merger. I thought of on-premises CDH clusters (and similarly HDP clusters) as trying to be the majority of your data infrastructure, but open so that it can integrate with other stuff. It's not just about having big data, but one place to store all of that data regardless of schema: massive database tables, logs, etc. all on shared hardware. AND frameworks to process it different ways in-place: SQL queries, Spark jobs, Search, etc. Data gravity was very important to the business model. As more people moved to the cloud, Hadoop-style storage was extremely expensive (naively moving your Hadoop cluster to 3x replication on EBS volumes would result in a nasty case of sticker shock) so the data would move to S3 / ADLS / GCP. And now you've lost your data gravity. Post-merger Cloudera focused less on on-premises clusters and tried to offer those same diverse workloads as a multi-cloud SaaS, with more focus on elasticity. This is hard because (a) there's a massive amount of surface area if you want enterprise customers to bring their own accounts, run all these managed open-source services in those accounts, and be multi-cloud, and (b) you're just competing more directly with the cloud vendors, on their turf as both a customer, partner and competitor.
  - threeseed 5y ago
    Would add that HDFS was a particular nightmare to manage. You had to worry about the size of files since the NameNode would be overloaded. Being a Java app running on the older JVMs it would do a full GC under heavy load and cause failovers. And it was impossible to get data in/out from outside the cluster using third party tools. I remember many companies seeing S3 and just being in shock that it was so cheap, limitless and that someone else was going to manage it all.
    - bpodgursky 5y ago
      It's interesting, because I think HDFS (and NameNodes in particular) were impressively engineered for a use-case which didn't quite materialize — ie, very fast metadata queries (they are still much faster than S3 API calls). Turns out that cheap, simple, and massively scalable object storage is just far far more important in practice. I think there are still a couple use-cases where HDFS dominates S3 (I think some HBase workloads?). But yeah, I scaled up and maintained a 2000+ Hadoop cluster for years, and I would never choose it over object storage if given any plausible alternative.

by wardb 5y ago

swyx 5y ago

for the uninitiated, do you mind diffing what Hortonworks used to do and what post merger Cloudera is now focused on?

threeseed 5y ago
Before the cloud took off Hortonworks and Cloudera owned the Big Data market. They both offered a Hadoop distribution but had different strengths e.g. Hortonworks had fine grained access control, Cloudera had a better SQL product with Impala. Then AWS came along and built their own which was significantly cheaper and more flexible as you could easily scale your cluster up/down. And so companies moved to it when they over time began to move to the cloud. The Hortonworks/Cloudera response to this threat was to put away their differences and merge together. Over time Big Data has evolved from being Hadoop centric to being much more ML/AI focused i.e. not just manipulating and querying the data but doing something interesting with it. And AWS, Azure, GCP have really jumped in with a whole suite of products that are tightly integrated with the rest of their cloud offerings. And it's a large part of what differentiates their offerings so they compete very hard. So Cloudera has no choice but to do things that cloud providers won't or can't do: (1) focus on non-cloud or multi-cloud and (2) offer a much more integrated and cohesive solution. But having spent 10+ years in this space and deployed many Hadoop clusters I can tell you that Cloudera is going to struggle. Companies that I never thought would move to the cloud e.g. banks are figuring out the security and regulatory challenges and eagerly moving across. And so it's going to be a Cloudera versus Amazon/Google/Microsoft which is an impossible fight.
- swyx 5y ago
  thank you! i have no history in this space so can't ask followups except to observe that the tendency of ML/AI to reward the "big gets bigger" phenomenon is exemplified here. I don't feel too great about that but also don't have ideas for a better system.
- wardb 5y ago
  Competing with Amazon/Google/Microsoft on their own cloud is...ehm...good luck with that indeed. I believe they should have partnered with them early on (real partnership, like a premier offering, not the rubber stamp / marketplace partnership).
  - manigandham 5y ago
    It can work, as that's exactly what Snowflake has done and it's one of the fastest-growing SaaS companies today. A good product is more valuable than a partnership.
    - 8ytecoder 5y ago
      Databricks as well. Azure sells both their own version and “Azure Databricks”
    - wardb 5y ago
      Good products, but don't discount their go-to-market strategy. IIRC MS/Azure is an early Databricks investor and their sales folks were heavily incentivised to sell it. They also pushed Snowflake in the early days until they had a competing product and their relationship status was upgraded to 'it's complicated'.
- bostonsre 5y ago
  AWS EMR is still pretty pricey compared to free ambari/cloudera running on ec2. Although, there is a lot of time and effort that needed to be put into automation that uses those ambari/cloudera hadoop management layers. After they merged, they got really aggressive and made moves that effectively killed each of the free versions. They definitely put another nail in the coffin of hadoop. Spark on kubernetes is pretty gorgeous and has been a successful route out of pricey hadoop infrastructure for my company.
macksd 5y ago
Worked at Cloudera pre- and post-merger. I thought of on-premises CDH clusters (and similarly HDP clusters) as trying to be the majority of your data infrastructure, but open so that it can integrate with other stuff. It's not just about having big data, but one place to store all of that data regardless of schema: massive database tables, logs, etc. all on shared hardware. AND frameworks to process it different ways in-place: SQL queries, Spark jobs, Search, etc. Data gravity was very important to the business model. As more people moved to the cloud, Hadoop-style storage was extremely expensive (naively moving your Hadoop cluster to 3x replication on EBS volumes would result in a nasty case of sticker shock) so the data would move to S3 / ADLS / GCP. And now you've lost your data gravity. Post-merger Cloudera focused less on on-premises clusters and tried to offer those same diverse workloads as a multi-cloud SaaS, with more focus on elasticity. This is hard because (a) there's a massive amount of surface area if you want enterprise customers to bring their own accounts, run all these managed open-source services in those accounts, and be multi-cloud, and (b) you're just competing more directly with the cloud vendors, on their turf as both a customer, partner and competitor.
- threeseed 5y ago
  Would add that HDFS was a particular nightmare to manage. You had to worry about the size of files since the NameNode would be overloaded. Being a Java app running on the older JVMs it would do a full GC under heavy load and cause failovers. And it was impossible to get data in/out from outside the cluster using third party tools. I remember many companies seeing S3 and just being in shock that it was so cheap, limitless and that someone else was going to manage it all.
  - bpodgursky 5y ago
    It's interesting, because I think HDFS (and NameNodes in particular) were impressively engineered for a use-case which didn't quite materialize — ie, very fast metadata queries (they are still much faster than S3 API calls). Turns out that cheap, simple, and massively scalable object storage is just far far more important in practice. I think there are still a couple use-cases where HDFS dominates S3 (I think some HBase workloads?). But yeah, I scaled up and maintained a 2000+ Hadoop cluster for years, and I would never choose it over object storage if given any plausible alternative.