12 ms·

Cloudera taken private for $5.3B, acquires Datacoral and Cazena

153 points by swyx 5y ago

erhserhdfd 5y ago
This is an interesting move. Many companies are still operating on-prem, hybrid cloud or they need to have a multicloud strategy (to operate in certain geographies or to avoid vendor lock-in). If Cloudera can get to the point where they have a big data infrastructure that is competitive with the native offerings from GCP, AWS, etc but easily supports on-prem, hybrid-cloud and multi-cloud, I would imagine there would be many ready corporate customers. Going private could give them the opportunity to make significant RnD investments away from the quarterly demands of a public company. On the other hand, the private equity backers could enact a bunch of cost-cutting, gut the company, cause its top employees to leave and load the company up with debt. It will be interesting to see how this shakes out.
- streetcat1 5y ago
  Well. Not. PE usually is about discipline. I.e. cash based discipline. So PE change the debt/equity ratio from 30/70 to 70/30, thus enforcing the company to be much more cash efficient. I.e. take LESS risks. To sum up, I do not envision more R & D.
  - markus_zhang 5y ago
    I think the problem of PE is 1) They need some quick return, 2) They don't necessarily understand the details of the tech (although they probably have access to some talents who do, or claim to do. So usually they just cut cost and make profit from it.
    - qeternity 5y ago
      I hear this argument a lot but with all due respect, simply understanding the tech is what got them into this issue. They didn’t understand the commercial. PE has some scars, but they don’t typically take over healthy, well run companies. They take over mismanaged companies, and refocus on the pursuit of profit that you seem to take issue with, ultimately saving the company from bankruptcy.
      - lazide 5y ago
        Eh, you had me up until the ‘ultimately saving from bankruptcy’ part. PE cares about investor return, not some high and mighty altruistic end goal no? There are plenty of PE deals of companies no where near insolvency, where the companies just had some more juice that could be squeezed (short/medium/long term depending, but usually short/medium term). If it’s a problem if THAT ends in bankruptcy or not depends on if they are able to get sufficient positive return/exit before they get hit by any losses - seen that many times.
        qeternity 5y ago
        How is saving from bankruptcy altruistic? It’s a financial outcome… If you can save a company from bankruptcy, that’s wealth that would otherwise have been destroyed.
        lazide 5y ago
        Because that isn’t the goal and is often not what happens? It is purchased to make return, not to save them from bankruptcy. If it is a side effect, that’s fine, but that is not why it’s done or what the goal is no?
jmspring 5y ago
Cloudera / Horton works merger -> President steps down -> Cloudera gets taken private. Interesting trajectory over the last few years.
- ternaryoperator 5y ago
  Agreed. It's astonishing, to me at least, that a company that was on the vanguard of both the cloud and big data was so incapable of figuring out a way to make real money off either one. Their sale today, despite the buyer premium, is well below their IPO price.
  - jmspring 5y ago
    One thing to note, and this was seen with the Skype purchase, typically contracts as part of a private equity deal limit stock issued to lower level employees but also generally have an aggressive clawback clause. I'm too lazy to find the best reference, but this is one re: Skype - https://www.businessinsider.com/skype-scandal-silver-lake-2011-6 https://www.businessinsider.com/skype-scandal-silver-lake-20...
    - t0mas88 5y ago
      The Skype one sounds like it wasn't such a weird deal. Those that got fired were rumoured to not get their options but in reality they did. Only someone that decided to quit voluntarily 13 months in didn't get an equity deal on the exit that happened later. And this wasn't a secret clause somewhere, those were the terms he had from the beginning.
  - Traster 5y ago
    Is this surprising? I wouldn't be surprised if the value in cloud accretes almost exclusively to the big cloud companies. Exit strategies for these kind of enterprises should primarily be "get acquired by X cloud provider".
    - paulryanrogers 5y ago
zx2391 5y ago
I was under the impression that private equity was mainly a vehicle for the rich to get richer. Serious question: Do companies have no choice but to go this route or why are they doing this?
- erhserhdfd 5y ago
  Many struggling technology companies will go private to all them to substantially invest in RnD. This allows then to make those expenditures without having the overhead and distractions of having to be a public company where investors are expecting short-term returns. Further, these private equity companies may then roll up several related businesses in an attempt to create synergies through sales or product. Ultimate software is a recent example of this: https://www.forbes.com/sites/antoinegara/2019/03/01/an-insiders-take-on-ultimate-softwares-11-billion-private-equity-takeover/?sh=68534574f9f5 https://www.forbes.com/sites/antoinegara/2019/03/01/an-insid...
  - bpodgursky 5y ago
    Yeah, I think there's a pretty big difference between tech companies going private (often to double-down R&D investment) and mature consumer brands going private (which is usually a way to extract profits and milk the carcass dry, see: Toys r Us, Olive Garden).
    - skeeter2020 5y ago
      this +100. You see the later in manture enterprise software a lot more these days and it is not fun to be a part of it. You get to watch software development transition from a profit generator to a cost center, which changes the game completely.
- jeffbee 5y ago
  The people with the authority to make the deal get paid huge sums for pulling the trigger. After that it doesn't matter to them that the company gets digested from the inside out.
- 5y ago
cosmodisk 5y ago
To all Cloudera's clients: pack your stuff and try to get an alternative vendor. Private Equity will not bring anything valuable to this business except higher prices,more aggressive sales and poor customer service.
- htrp 5y ago
  This 100%
- bastardoperator 5y ago
  Exactly this, don't get squeezed.
- bostonsre 5y ago
  After the merger with hortonworks, they already went extremely aggressive and I didn't think it was possible for them to have an even dimmer future...
- thirsteh 5y ago
  Serious question: Has PE M&A ever led to an improved product? (Maybe that's just a stupid rather than a serious question)
  - dimitrios1 5y ago
    Dell? Silver Lake played a big role in that IIRC.
    - pinewurst 5y ago
      Silver Lake was only a source for money, not “management expertise” on that deal.
      - dimitrios1 5y ago
        Having worked for a Silver Lake funded company (I originally called a startup, but that's not fair to say anymore for a private company that now makes billions), I can assure you that they don't take a back seat to how the company is ran (that's not to say they take a direct hands on approach, either).
flakiness 5y ago
For a reference: Hadoop on Google Trends peeked at 2015: https://trends.google.com/trends/explore?date=all&geo=US&q=hadoop https://trends.google.com/trends/explore?date=all&geo=US&q=h... "Hadoop is dead" seems to be a popular topic in past few years. https://www.google.com/search?q=hadoop+is+dead https://www.google.com/search?q=hadoop+is+dead
- asperous 5y ago
  Good points of evidence, however you can't read too much into the google trends. A lot of technologies peak when they are new and then fall to a steady state. For example "computers" peaked pre-2004: https://trends.google.com/trends/explore?date=all&geo=US&q=computers https://trends.google.com/trends/explore?date=all&geo=US&q=c... Javascript: https://trends.google.com/trends/explore?date=all&geo=US&q=javascript https://trends.google.com/trends/explore?date=all&geo=US&q=j... Machine learning: https://trends.google.com/trends/explore?date=all&geo=US&q=machine%20learning https://trends.google.com/trends/explore?date=all&geo=US&q=m...
  - threeseed 5y ago
    In this case it's accurate. Hadoop is largely dead. YARN, Hive, HDFS, MapReduce have been replaced by Kubernetes, Snowflake, S3, Spark.
    - Fordec 5y ago
      And even that will continue to change. Kubernetes is overused right now, it has its place but it's not nearly universally the right tool for the job. Snowflake will eventually fall to something else due it's poor economics. S3 and Spark though I anticipate to be around for a good few years and if they lose out it will be to imitators or evolutionary equivalents.
      - peterthehacker 5y ago
        > Snowflake will eventually fall to something else due it's poor economics. Can you elaborate on this point? What’s wrong with their model?
wardb 5y ago
Worked at Hortonworks and post merger Cloudera. Was interesting to see how market demand changed over the years and how the company worked on re-inventing themselves. Databricks and Snowflake seemed to understand SaaS earlier and better though. Still have some great friends working at Cloudera, and hope this will indeed accelerate a great next phase.
- swyx 5y ago
  for the uninitiated, do you mind diffing what Hortonworks used to do and what post merger Cloudera is now focused on?
  - threeseed 5y ago
    Before the cloud took off Hortonworks and Cloudera owned the Big Data market. They both offered a Hadoop distribution but had different strengths e.g. Hortonworks had fine grained access control, Cloudera had a better SQL product with Impala. Then AWS came along and built their own which was significantly cheaper and more flexible as you could easily scale your cluster up/down. And so companies moved to it when they over time began to move to the cloud. The Hortonworks/Cloudera response to this threat was to put away their differences and merge together. Over time Big Data has evolved from being Hadoop centric to being much more ML/AI focused i.e. not just manipulating and querying the data but doing something interesting with it. And AWS, Azure, GCP have really jumped in with a whole suite of products that are tightly integrated with the rest of their cloud offerings. And it's a large part of what differentiates their offerings so they compete very hard. So Cloudera has no choice but to do things that cloud providers won't or can't do: (1) focus on non-cloud or multi-cloud and (2) offer a much more integrated and cohesive solution. But having spent 10+ years in this space and deployed many Hadoop clusters I can tell you that Cloudera is going to struggle. Companies that I never thought would move to the cloud e.g. banks are figuring out the security and regulatory challenges and eagerly moving across. And so it's going to be a Cloudera versus Amazon/Google/Microsoft which is an impossible fight.
    - swyx 5y ago
      thank you! i have no history in this space so can't ask followups except to observe that the tendency of ML/AI to reward the "big gets bigger" phenomenon is exemplified here. I don't feel too great about that but also don't have ideas for a better system.
andyxor 5y ago
AWS, GCP and Azure ate Cloudera's lunch.
- bombcar 5y ago
  A huge portion of the growth in cloud was the relatively cheaper aspect for a VPS vs places that sold physical server rackspace. Once the foothold was in there the value-add of services on top became a huge revenue generator.
markus_zhang 5y ago
Clayton, Dubilier & Rice (CD&R) and KKR. When I see the name of KKR mentioned...
- kolbe 5y ago
  I don't even know what they do anymore. They used to make money by just raiding their portfolio companies' pensions, but now that corporations don't offer private pensions, I just don't know. Maybe they're actually improving the companies?
ralph84 5y ago
The elephant (no pun intended) in the room continues to be most data in the enterprise isn't big data. You don't need mapreduce when your data set fits in RAM.
- peterthehacker 5y ago
  Yes! And if it doesn’t fit in RAM, then most businesses will only need an OLAP data warehouse, like Snowflake or Redshift.
  - threeseed 5y ago
    And if you want to join data from different SaaS and internal systems e.g. Google Analytics and a Pega decisioning system. Are you going to spend months upfront carefully modelling the data in order to ingest it making sure to handle schema and DQ issues etc. All to support one use case who only needs a handful of fields. No. Which is why data lakes exist. Because it's cost effective. You simply dump the data and ask the Engineer or Data Scientist building the use case to do the heavy lifting rather than a centralised data team.
    - peterthehacker 5y ago
      There are integration companies that solve this specific use-case. I’ve used Fivetran [0] and highly recommend it. They will extract-load data from your SaaS to your warehouse and your data scientists can run SQL against the tables. Their most popular warehouses are Redshift and Snowflake. So you can still use a centralized data warehouse without dedicating internal resources to the integrations. [0] https://fivetran.com/ https://fivetran.com/
      - ak217 5y ago
        What I find amazing is that Fivetran is a bunch of glue code to forward data between different APIs and database formats, and it's legitimately useful, in part because when the upstream API breaks they go and fix the connectors for you instead of you having to deal with the resulting emergency... but it's only needed because data interchange standards are in such poor shape. If users demanded that SaaS products make data/event streams/replication logs available via robust and standardized APIs, a lot of the use cases for Fivetran would disappear.
dgudkov 5y ago
>After all, we invented the whole idea of Big Data. Really?
iblaine 5y ago
I was a Cloudera client in 2015. I've since moved to company where we did on prem (thousands of nodes) then another AWS shop. Each case was an example of a superior solution to Cloudera. Cloudera's market seems to be shrinking & they need to change. On Datacoral, it seems like a very good product, simplifying the overhead of data pipeline development but still staying modern. Datacoral has many cool features, such as treating ETLs as code with an option to develop via GUI. The industry is moving that way, has been for some time. Good for Datacoral to get acquired and get exposure to so many new customers. But Cloudera has their work cut out for them to stay relevant in a market that is extremely competitive.
lmeyerov 5y ago
Been quite a ride to watch, feels like a surprise every couple of years! Looking ahead, on the one side, we're seeing a privately hosted (on-prem + cloud) revival due to multicloud / lockin distrust + extreme cloud markup for AI/ML (GPUs etc) on the other, and for cloud, insane 30-50% YoY growth rates even at $B scales. On the other, I'm not sure how that lines up w/ Cloudera's current products and the investment team's R&D/M&A appetite It also makes me wonder what is the value of being in n-th place. We're reviewing which new connectors to prioritize for our own viz tool dev, and cloud-native offerings like snowflake/databricks/redshift are generally ahead of on-prem ones, and we're not the only ones. So what's the value of current growth, and of paying premiums to leap frog (ML, GPUs, etc.), esp. when you have the kind of footprint cloudera earned?
alphachloride 5y ago
One annoying thing I have noticed about many company/project blogs that they have no link on the header to go to the actual project website (project.com). Instead the main logo takes back to the blog (blog.project.com). You have to manually type the URL for the actual website. For e.g. even this site: https://blog.ycombinator.com/ https://blog.ycombinator.com/
- nwsm 5y ago
  The HN Algoila page also doesn't link back to HN: https://hn.algolia.com/ https://hn.algolia.com/
peter303 5y ago
KJR! Expect half your staff to be gone next year after massive cost cutting and leveraging.
dikei 5y ago
Cloudera and Hortonwork promise an easy way to deploy on-premise Hadoop clusters. Yet, HDP & CDH are still too hard to maintain for the majority of companies: * Tuning for HDFS and HBase is hard. Too many files and GC pause will kill you. Hive Metastore is also a bottleneck. * It's a nightmare to figure out how to configure some third-party software that's outside the bundle. * It's even harder, if you want to use a more up-to-date version of a software included in the bundle. We still haven't figured out how to use Spark 3 with our old HDP clusters. * Sometime the included software is a buggy mess. For example, HDP 3 provides HBase 2.0, which is absolutely unusable. Luckily, we were able to migrate to HBase 2.2 before putting too much data into it. All in all, it still takes a lot of ops work to own a Cloudera's Hadoop clusters. I can't wait until the time when we can move to a better solution.
- pm90 5y ago
  This might be a silly question but: Why is it so hard to use? What made cloudera not see how difficult their product was to use and invest in fixing that? It appears to have costed them the market, yes?
  - nl 5y ago
    > Why is it so hard to use? What made cloudera not see how difficult their product was to use and invest in fixing that? It is packaged open source software, from multiple projects (some of which are financed by Cloudera) all of which release at different paces. And distributed systems are really hard to get right. There are lots of knobs to tune, all of which are needed in some circumstances, but.. there are lots of things to go wrong. > It appears to have costed them the market, yes? Cloudera is probably the market leader for on-prem Hadoop clusters. This is a big market (even now - lots of defence clients who have issues with cloud). But generally this market is getting eaten by Cloud SAAS products.
  - dikei 5y ago
    TL DR: Distributed system is hard, and Hadoop cluster are not really designed for a dynamic environment. There are as probably as many services running inside a Hadoop clusters as a Micro Service meshes, except: * The services can be huge: HDFS Namenode can takes hundreds GB of RAM, hours to start up in a multi PB cluster. Updating configuration requires restarting, so it is a huge pain if you care about zero downtime. * The services are often bound to host and can not be easily migrated to other hosts. * The communication interfaces between services are not well-defined: Hive Metastore, for example, didn't have a formal protocol documentation for a long time, yet everything depends on it to store metadata (I'm not even sure if they have the protocol now). As a result, services are often tightly coupled: you need everything to be compiled with the same libraries version, or stuffs might not work correctly. Furthermore, due to dynamic code loading, issues might not surface until days later - in the middle of the night, making life miserable for everyone.
colin_mccabe 5y ago
I worked at Cloudera in the early 2010s, before they went IPO. It's a bit sad to see the way things played out. All I can say is, it was a fun ride while it lasted. Cloudera actually started in the cloud, but quickly moved to selling mostly on-premise software. Back then, cloud was a lot less cost competitive (it still is not really cost competitive) https://a16z.com/2021/05/27/cost-of-cloud-paradox-market-cap-cloud-lifecycle-scale-growth-repatriation-optimization/ https://a16z.com/2021/05/27/cost-of-cloud-paradox-market-cap... The on-premise business had much higher margins, which kind of "took the oxygen out of the room" for the cloud side of the business. In retrospect, neglecting the cloud market was a big mistake, of course. Cloudera management knew even back then that Hadoop was eventually going to be superseded by a newer technology. We tried to build that technology, in the form of Impala, an optimized SQL query engine. Unfortunately, it never really took off in the marketplace, for several reasons. For one, its dialect of SQL was not compatible with Hive, the traditional SQL frontend for Hadoop. For another, early versions of Impala were somewhat unstable, and didn't have the fault-tolerance of traditional Hadoop tools. From my perspective, the main issue that cloud solves is that managing the server side stuff is really hard, and requires experts. Cloud also aligns expectations: it's really clear to both customer and vendor that a cloud contract is not a stepping stone towards self-managing. Unfortunately, Cloudera just didn't see the shift towards cloud coming, and they're now paying the price.

153 points by swyx 5y ago

This is an interesting move. Many companies are still operating on-prem, hybrid cloud or they need to have a multicloud strategy (to operate in certain geographies or to avoid vendor lock-in). If Cloudera can get to the point where they have a big data infrastructure that is competitive with the native offerings from GCP, AWS, etc but easily supports on-prem, hybrid-cloud and multi-cloud, I would imagine there would be many ready corporate customers. Going private could give them the opportunity to make significant RnD investments away from the quarterly demands of a public company. On the other hand, the private equity backers could enact a bunch of cost-cutting, gut the company, cause its top employees to leave and load the company up with debt. It will be interesting to see how this shakes out.

streetcat1 5y ago
Well. Not. PE usually is about discipline. I.e. cash based discipline. So PE change the debt/equity ratio from 30/70 to 70/30, thus enforcing the company to be much more cash efficient. I.e. take LESS risks. To sum up, I do not envision more R & D.
- markus_zhang 5y ago
  I think the problem of PE is 1) They need some quick return, 2) They don't necessarily understand the details of the tech (although they probably have access to some talents who do, or claim to do. So usually they just cut cost and make profit from it.
  - qeternity 5y ago
    I hear this argument a lot but with all due respect, simply understanding the tech is what got them into this issue. They didn’t understand the commercial. PE has some scars, but they don’t typically take over healthy, well run companies. They take over mismanaged companies, and refocus on the pursuit of profit that you seem to take issue with, ultimately saving the company from bankruptcy.
    - lazide 5y ago
      Eh, you had me up until the ‘ultimately saving from bankruptcy’ part. PE cares about investor return, not some high and mighty altruistic end goal no? There are plenty of PE deals of companies no where near insolvency, where the companies just had some more juice that could be squeezed (short/medium/long term depending, but usually short/medium term). If it’s a problem if THAT ends in bankruptcy or not depends on if they are able to get sufficient positive return/exit before they get hit by any losses - seen that many times.
      - qeternity 5y ago
        How is saving from bankruptcy altruistic? It’s a financial outcome… If you can save a company from bankruptcy, that’s wealth that would otherwise have been destroyed.
        lazide 5y ago
        Because that isn’t the goal and is often not what happens? It is purchased to make return, not to save them from bankruptcy. If it is a side effect, that’s fine, but that is not why it’s done or what the goal is no?

jmspring 5y ago

Cloudera / Horton works merger -> President steps down -> Cloudera gets taken private. Interesting trajectory over the last few years.

ternaryoperator 5y ago
Agreed. It's astonishing, to me at least, that a company that was on the vanguard of both the cloud and big data was so incapable of figuring out a way to make real money off either one. Their sale today, despite the buyer premium, is well below their IPO price.
- jmspring 5y ago
  One thing to note, and this was seen with the Skype purchase, typically contracts as part of a private equity deal limit stock issued to lower level employees but also generally have an aggressive clawback clause. I'm too lazy to find the best reference, but this is one re: Skype - https://www.businessinsider.com/skype-scandal-silver-lake-2011-6 https://www.businessinsider.com/skype-scandal-silver-lake-20...
  - t0mas88 5y ago
    The Skype one sounds like it wasn't such a weird deal. Those that got fired were rumoured to not get their options but in reality they did. Only someone that decided to quit voluntarily 13 months in didn't get an equity deal on the exit that happened later. And this wasn't a secret clause somewhere, those were the terms he had from the beginning.
- Traster 5y ago
  Is this surprising? I wouldn't be surprised if the value in cloud accretes almost exclusively to the big cloud companies. Exit strategies for these kind of enterprises should primarily be "get acquired by X cloud provider".
  - paulryanrogers 5y ago

zx2391 5y ago

I was under the impression that private equity was mainly a vehicle for the rich to get richer. Serious question: Do companies have no choice but to go this route or why are they doing this?

erhserhdfd 5y ago
Many struggling technology companies will go private to all them to substantially invest in RnD. This allows then to make those expenditures without having the overhead and distractions of having to be a public company where investors are expecting short-term returns. Further, these private equity companies may then roll up several related businesses in an attempt to create synergies through sales or product. Ultimate software is a recent example of this: https://www.forbes.com/sites/antoinegara/2019/03/01/an-insiders-take-on-ultimate-softwares-11-billion-private-equity-takeover/?sh=68534574f9f5 https://www.forbes.com/sites/antoinegara/2019/03/01/an-insid...
- bpodgursky 5y ago
  Yeah, I think there's a pretty big difference between tech companies going private (often to double-down R&D investment) and mature consumer brands going private (which is usually a way to extract profits and milk the carcass dry, see: Toys r Us, Olive Garden).
  - skeeter2020 5y ago
    this +100. You see the later in manture enterprise software a lot more these days and it is not fun to be a part of it. You get to watch software development transition from a profit generator to a cost center, which changes the game completely.
jeffbee 5y ago
The people with the authority to make the deal get paid huge sums for pulling the trigger. After that it doesn't matter to them that the company gets digested from the inside out.
5y ago

cosmodisk 5y ago

To all Cloudera's clients: pack your stuff and try to get an alternative vendor. Private Equity will not bring anything valuable to this business except higher prices,more aggressive sales and poor customer service.

htrp 5y ago
This 100%
bastardoperator 5y ago
Exactly this, don't get squeezed.
bostonsre 5y ago
After the merger with hortonworks, they already went extremely aggressive and I didn't think it was possible for them to have an even dimmer future...
thirsteh 5y ago
Serious question: Has PE M&A ever led to an improved product? (Maybe that's just a stupid rather than a serious question)
- dimitrios1 5y ago
  Dell? Silver Lake played a big role in that IIRC.
  - pinewurst 5y ago
    Silver Lake was only a source for money, not “management expertise” on that deal.
    - dimitrios1 5y ago
      Having worked for a Silver Lake funded company (I originally called a startup, but that's not fair to say anymore for a private company that now makes billions), I can assure you that they don't take a back seat to how the company is ran (that's not to say they take a direct hands on approach, either).

flakiness 5y ago

For a reference: Hadoop on Google Trends peeked at 2015: https://trends.google.com/trends/explore?date=all&geo=US&q=hadoop https://trends.google.com/trends/explore?date=all&geo=US&q=h... "Hadoop is dead" seems to be a popular topic in past few years. https://www.google.com/search?q=hadoop+is+dead https://www.google.com/search?q=hadoop+is+dead

asperous 5y ago
Good points of evidence, however you can't read too much into the google trends. A lot of technologies peak when they are new and then fall to a steady state. For example "computers" peaked pre-2004: https://trends.google.com/trends/explore?date=all&geo=US&q=computers https://trends.google.com/trends/explore?date=all&geo=US&q=c... Javascript: https://trends.google.com/trends/explore?date=all&geo=US&q=javascript https://trends.google.com/trends/explore?date=all&geo=US&q=j... Machine learning: https://trends.google.com/trends/explore?date=all&geo=US&q=machine%20learning https://trends.google.com/trends/explore?date=all&geo=US&q=m...
- threeseed 5y ago
  In this case it's accurate. Hadoop is largely dead. YARN, Hive, HDFS, MapReduce have been replaced by Kubernetes, Snowflake, S3, Spark.
  - Fordec 5y ago
    And even that will continue to change. Kubernetes is overused right now, it has its place but it's not nearly universally the right tool for the job. Snowflake will eventually fall to something else due it's poor economics. S3 and Spark though I anticipate to be around for a good few years and if they lose out it will be to imitators or evolutionary equivalents.
    - peterthehacker 5y ago
      > Snowflake will eventually fall to something else due it's poor economics. Can you elaborate on this point? What’s wrong with their model?

wardb 5y ago

Worked at Hortonworks and post merger Cloudera. Was interesting to see how market demand changed over the years and how the company worked on re-inventing themselves. Databricks and Snowflake seemed to understand SaaS earlier and better though. Still have some great friends working at Cloudera, and hope this will indeed accelerate a great next phase.

swyx 5y ago
for the uninitiated, do you mind diffing what Hortonworks used to do and what post merger Cloudera is now focused on?
- threeseed 5y ago
  Before the cloud took off Hortonworks and Cloudera owned the Big Data market. They both offered a Hadoop distribution but had different strengths e.g. Hortonworks had fine grained access control, Cloudera had a better SQL product with Impala. Then AWS came along and built their own which was significantly cheaper and more flexible as you could easily scale your cluster up/down. And so companies moved to it when they over time began to move to the cloud. The Hortonworks/Cloudera response to this threat was to put away their differences and merge together. Over time Big Data has evolved from being Hadoop centric to being much more ML/AI focused i.e. not just manipulating and querying the data but doing something interesting with it. And AWS, Azure, GCP have really jumped in with a whole suite of products that are tightly integrated with the rest of their cloud offerings. And it's a large part of what differentiates their offerings so they compete very hard. So Cloudera has no choice but to do things that cloud providers won't or can't do: (1) focus on non-cloud or multi-cloud and (2) offer a much more integrated and cohesive solution. But having spent 10+ years in this space and deployed many Hadoop clusters I can tell you that Cloudera is going to struggle. Companies that I never thought would move to the cloud e.g. banks are figuring out the security and regulatory challenges and eagerly moving across. And so it's going to be a Cloudera versus Amazon/Google/Microsoft which is an impossible fight.
  - swyx 5y ago
    thank you! i have no history in this space so can't ask followups except to observe that the tendency of ML/AI to reward the "big gets bigger" phenomenon is exemplified here. I don't feel too great about that but also don't have ideas for a better system.

andyxor 5y ago

AWS, GCP and Azure ate Cloudera's lunch.

bombcar 5y ago
A huge portion of the growth in cloud was the relatively cheaper aspect for a VPS vs places that sold physical server rackspace. Once the foothold was in there the value-add of services on top became a huge revenue generator.

markus_zhang 5y ago

Clayton, Dubilier & Rice (CD&R) and KKR. When I see the name of KKR mentioned...

kolbe 5y ago
I don't even know what they do anymore. They used to make money by just raiding their portfolio companies' pensions, but now that corporations don't offer private pensions, I just don't know. Maybe they're actually improving the companies?

ralph84 5y ago

The elephant (no pun intended) in the room continues to be most data in the enterprise isn't big data. You don't need mapreduce when your data set fits in RAM.

peterthehacker 5y ago
Yes! And if it doesn’t fit in RAM, then most businesses will only need an OLAP data warehouse, like Snowflake or Redshift.
- threeseed 5y ago
  And if you want to join data from different SaaS and internal systems e.g. Google Analytics and a Pega decisioning system. Are you going to spend months upfront carefully modelling the data in order to ingest it making sure to handle schema and DQ issues etc. All to support one use case who only needs a handful of fields. No. Which is why data lakes exist. Because it's cost effective. You simply dump the data and ask the Engineer or Data Scientist building the use case to do the heavy lifting rather than a centralised data team.
  - peterthehacker 5y ago
    There are integration companies that solve this specific use-case. I’ve used Fivetran [0] and highly recommend it. They will extract-load data from your SaaS to your warehouse and your data scientists can run SQL against the tables. Their most popular warehouses are Redshift and Snowflake. So you can still use a centralized data warehouse without dedicating internal resources to the integrations. [0] https://fivetran.com/ https://fivetran.com/
    - ak217 5y ago
      What I find amazing is that Fivetran is a bunch of glue code to forward data between different APIs and database formats, and it's legitimately useful, in part because when the upstream API breaks they go and fix the connectors for you instead of you having to deal with the resulting emergency... but it's only needed because data interchange standards are in such poor shape. If users demanded that SaaS products make data/event streams/replication logs available via robust and standardized APIs, a lot of the use cases for Fivetran would disappear.

dgudkov 5y ago

>After all, we invented the whole idea of Big Data. Really?

iblaine 5y ago

I was a Cloudera client in 2015. I've since moved to company where we did on prem (thousands of nodes) then another AWS shop. Each case was an example of a superior solution to Cloudera. Cloudera's market seems to be shrinking & they need to change. On Datacoral, it seems like a very good product, simplifying the overhead of data pipeline development but still staying modern. Datacoral has many cool features, such as treating ETLs as code with an option to develop via GUI. The industry is moving that way, has been for some time. Good for Datacoral to get acquired and get exposure to so many new customers. But Cloudera has their work cut out for them to stay relevant in a market that is extremely competitive.

lmeyerov 5y ago

Been quite a ride to watch, feels like a surprise every couple of years! Looking ahead, on the one side, we're seeing a privately hosted (on-prem + cloud) revival due to multicloud / lockin distrust + extreme cloud markup for AI/ML (GPUs etc) on the other, and for cloud, insane 30-50% YoY growth rates even at $B scales. On the other, I'm not sure how that lines up w/ Cloudera's current products and the investment team's R&D/M&A appetite It also makes me wonder what is the value of being in n-th place. We're reviewing which new connectors to prioritize for our own viz tool dev, and cloud-native offerings like snowflake/databricks/redshift are generally ahead of on-prem ones, and we're not the only ones. So what's the value of current growth, and of paying premiums to leap frog (ML, GPUs, etc.), esp. when you have the kind of footprint cloudera earned?

alphachloride 5y ago

One annoying thing I have noticed about many company/project blogs that they have no link on the header to go to the actual project website (project.com). Instead the main logo takes back to the blog (blog.project.com). You have to manually type the URL for the actual website. For e.g. even this site: https://blog.ycombinator.com/ https://blog.ycombinator.com/

nwsm 5y ago
The HN Algoila page also doesn't link back to HN: https://hn.algolia.com/ https://hn.algolia.com/

peter303 5y ago

KJR! Expect half your staff to be gone next year after massive cost cutting and leveraging.

dikei 5y ago

Cloudera and Hortonwork promise an easy way to deploy on-premise Hadoop clusters. Yet, HDP & CDH are still too hard to maintain for the majority of companies: * Tuning for HDFS and HBase is hard. Too many files and GC pause will kill you. Hive Metastore is also a bottleneck. * It's a nightmare to figure out how to configure some third-party software that's outside the bundle. * It's even harder, if you want to use a more up-to-date version of a software included in the bundle. We still haven't figured out how to use Spark 3 with our old HDP clusters. * Sometime the included software is a buggy mess. For example, HDP 3 provides HBase 2.0, which is absolutely unusable. Luckily, we were able to migrate to HBase 2.2 before putting too much data into it. All in all, it still takes a lot of ops work to own a Cloudera's Hadoop clusters. I can't wait until the time when we can move to a better solution.

pm90 5y ago
This might be a silly question but: Why is it so hard to use? What made cloudera not see how difficult their product was to use and invest in fixing that? It appears to have costed them the market, yes?
- nl 5y ago
  > Why is it so hard to use? What made cloudera not see how difficult their product was to use and invest in fixing that? It is packaged open source software, from multiple projects (some of which are financed by Cloudera) all of which release at different paces. And distributed systems are really hard to get right. There are lots of knobs to tune, all of which are needed in some circumstances, but.. there are lots of things to go wrong. > It appears to have costed them the market, yes? Cloudera is probably the market leader for on-prem Hadoop clusters. This is a big market (even now - lots of defence clients who have issues with cloud). But generally this market is getting eaten by Cloud SAAS products.
- dikei 5y ago
  TL DR: Distributed system is hard, and Hadoop cluster are not really designed for a dynamic environment. There are as probably as many services running inside a Hadoop clusters as a Micro Service meshes, except: * The services can be huge: HDFS Namenode can takes hundreds GB of RAM, hours to start up in a multi PB cluster. Updating configuration requires restarting, so it is a huge pain if you care about zero downtime. * The services are often bound to host and can not be easily migrated to other hosts. * The communication interfaces between services are not well-defined: Hive Metastore, for example, didn't have a formal protocol documentation for a long time, yet everything depends on it to store metadata (I'm not even sure if they have the protocol now). As a result, services are often tightly coupled: you need everything to be compiled with the same libraries version, or stuffs might not work correctly. Furthermore, due to dynamic code loading, issues might not surface until days later - in the middle of the night, making life miserable for everyone.

colin_mccabe 5y ago

I worked at Cloudera in the early 2010s, before they went IPO. It's a bit sad to see the way things played out. All I can say is, it was a fun ride while it lasted. Cloudera actually started in the cloud, but quickly moved to selling mostly on-premise software. Back then, cloud was a lot less cost competitive (it still is not really cost competitive) https://a16z.com/2021/05/27/cost-of-cloud-paradox-market-cap-cloud-lifecycle-scale-growth-repatriation-optimization/ https://a16z.com/2021/05/27/cost-of-cloud-paradox-market-cap... The on-premise business had much higher margins, which kind of "took the oxygen out of the room" for the cloud side of the business. In retrospect, neglecting the cloud market was a big mistake, of course. Cloudera management knew even back then that Hadoop was eventually going to be superseded by a newer technology. We tried to build that technology, in the form of Impala, an optimized SQL query engine. Unfortunately, it never really took off in the marketplace, for several reasons. For one, its dialect of SQL was not compatible with Hive, the traditional SQL frontend for Hadoop. For another, early versions of Impala were somewhat unstable, and didn't have the fault-tolerance of traditional Hadoop tools. From my perspective, the main issue that cloud solves is that managing the server side stuff is really hard, and requires experts. Cloud also aligns expectations: it's really clear to both customer and vendor that a cloud contract is not a stepping stone towards self-managing. Unfortunately, Cloudera just didn't see the shift towards cloud coming, and they're now paying the price.