13 ms·

MapR may shut down as investor pulls out after ‘extremely poor results’

108 points by heyyyouu 7y ago

Joeri 7y ago
So, that would mean the hadoop distro market would be down to just cloudera? A single vendor market is not a market, it's a legacy product.
- russtrotter 7y ago
  AFAIK, AWS's EMR and Glue leverage Hadoop under the covers. Does that count towards that market? and/or at least keeping Hadoop development active?
  - derefr 7y ago
    Hadoop ≠ Hadoop distro. Hadoop is fine and healthy; what is in question is whether larger repackagings of it with other vendor-specific components is an interesting product any more.
  - filesystem 7y ago
    Google's dataproc and Azure's HDInsight as well. The cloud players are swallowing whatever is left of the hadoop market.
  - threeseed 7y ago
    EMR is their managed distro of Hadoop. DataPipeline is their managed Hadoop for ETL. Glue ETL is their managed Spark. And it all absolutely counts towards the market.
- buryat 7y ago
  Everybody is moving to the cloud so EMR, Dataproc, Dataflow, etc. are still going to be around Plus everyone is moving away from Hadoop and HDFS, so it's Spark, S3, GCS, Databricks, etc
- badpun 7y ago
  Is Microsoft Office or Microsoft Windows a legacy product?
- nwsm 7y ago
  Azure's HDInsight all-in-one suite uses it.
  - zjaffee 7y ago
    HDInsite is built by hortonworks which was acquired by cloudera.
  - opportune 7y ago
    In addition to HDInsight being cloudera, it's also basically a legacy product. Azure technical solutions people are now all pushing databricks. And personally, when designing a new project, I don't see why you would use HDInsight instead of a bespoke distributed vm architecture with one (azure storage) or two-tier storage (one local to the vm, one in Azure storage) unless you really need to use some hadoop jar you found lying around.
- jamesblonde 7y ago
  Actually, there is another vendor. A new one - www.logicalclocks.com - we are selling a next-generation version of Hadoop, with consistent distributed metadata for HDFS, it's built on TLS/SSL (not kerberos), and it's laser focussed on Data Science. We have customers with racks with 10s of GPUs training models with 10s of TBs of feature data. And we also built the industry's first enterprise feature store for ML on this platform. References: https://www.logicalclocks.com/feature-store/ https://www.logicalclocks.com/feature-store/ https://www.logicalclocks.com/introducing-hops-hadoop/ https://www.logicalclocks.com/introducing-hops-hadoop/
kwillets 7y ago
MapR has been reduced. Hadoop was the floppy disk of big data. Ubiquitous, but always beat by other solutions.
- threeseed 7y ago
  This is simply not true. Hadoop is growing massively. All that is happening is that the vendors are being killed by the cloud.
  - spork12 7y ago
    I've seen Hadoop and HDFS dying off, and being replaced by Spark, S3, and Azure Storage. There is also a lot of good work being done by DataBricks in creating an easy to manage spark platform.
kod 7y ago
Hope they're able to open source some of their tech if this does come to pass.
notyourday 7y ago
To me, and a few people that I know, the most interesting part of MapR was their NFS server over multiple nodes. It solved a real problem that real customers (i.e. enterprises with money for whom tech made stuff work and not was work in itself) were willing to pay a lot of money for ( see Isilon, EMC, NetApp, etc ) but that was a non-sexy part of the MapR business -- Big Data part was much sexier. Over the last 5 years the real money making enterprises solved their "it should look like a big file system" problem so "the rest of our stuff works" issue stopped being an issue ( in a process Isilon got bought by the EMC, EMC got bought by Dell ) either by buying those specialized solutions, moving to object store or building home grown systems that worked with their specific applications and big data people went with newer, more shiny big data solutions, leaving MapR with no market.
- heyyyouu 7y ago
  Which of the shinier, newer solutions do you think people have found more attractive in particular? Genuinely curious, thanks!
  - notyourday 7y ago
    I'm not someone very much in the "big data" [0] but from what I see streaming analytics especially Spark-based solutions are eating the world that used to be dominated by Hadoop. I also think cloud based data lakes and tooling around them seriously decreased the appeal of Hadoop. [0] If I can fit your big data into my memory, it is not "big data".
    - brokensegue 7y ago
      i mean spark is often deployed on top of yarn where it's still in the "hadoop" sphere.
      - notyourday 7y ago
        But I'm not loading up on my SuperMicro, Dell or HP servers. So there's no need to buy a decent Hadoop implementation from MapR.
joehandzik 7y ago
We're seeing a lot of regret around sprawling Hadoop deployments, so this doesn't surprise me. Other Hadoop vendors (vendor?) pivoting to machine learning is a bandaid as the compute capabilities scale beyond HDFS's performance limitations. Look towards new-gen startups around NVME/NVMEOF (WekaIO, Excelero, E8, etc etc) to fill the void. The question is going to be: will anyone provide an intelligent way to maintain compatibility with applications that expect to interface with HDFS rather than POSIX? It's a bit of a gap right now from what we see.
- joehandzik 7y ago
  Note that apparently the layoffs have already happened. Engineers are already interviewing elsewhere.
- capkutay 7y ago
  Hadoop's whole appeal was it was cheap and scalable. Did it actually work without serious engineering teams maintaining each distribution? absolutely not. the hype was purely VC funded. fast forward 10 years and hadoop has basically been killed off by hosted storage services that are more expensive but 10000x easier to manage.
  - dmix 7y ago
    I'm not really familiar with this industry. Are these Hadoop clusters hosted locally by these businesses? And MapR et all were providing the software/consulting to help manage them? Then I'm guessing better cloud hosted options came out offering similar capabilities. If that's the case was it really a big surprise that "cloud" hosting would eat any self-hosted platform's lunch?
    - capkutay 7y ago
      yes mostly locally hosted back then. the idea was you'd have this massive distributed file system to hold all your data, and VC backed comopanies like Cloudera, MapR, Hortonworks promised everyone that they'd build the SQL layer, the data warehousing, BI, and all the other enterprise features you'd need to basically replace your expensive Teradata, Oracle Exadata, and other data warehousing systems. But of course it didn't play out that way.
bitL 7y ago
Weren't they killed the moment Spark appeared, having heavily invested in accelerating Hadoop? I am surprised that Cloudera is still around...
- SEJeff 7y ago
  And spark was sort of killed by Storm / Heron as even Spark Streaming does microbatches and isn't truly realtime.
  - Dunedan 7y ago
    Wasn't it the other way around? Wasn't Storm killed by Spark? At least once Spark was out and gained traction, nobody in my bubble talked about Storm anymore. But I guess it depends on the bubble you're in.
    - brokensegue 7y ago
      yeah their timeline seems backwards
  - suresk 7y ago
    Wouldn't Flink be more likely to replace Spark? I don't hear about a lot of people using Storm these days, but I guess a lot of this depends on whatever bubble you happen to be in.
  - capkutay 7y ago
    in what world was spark killed by storm/heron? last i checked the spark conference had over 10k attendees this year. storm is barely supported and heron never picked up steam.
  - threeseed 7y ago
    What ? Spark Streaming is just one part of Spark and the far less used part as well. Spark hasn't been killed by anything and is still heavily used for ETL and ML.
  - mruts 7y ago
purplezooey 7y ago
I think they'll be fine. At my last company we had a large MapR cluster and it was hands down more reliable and user-friendly than anything else. Maybe Cloudera will pick them up.
- jrg 7y ago
  We've used them as a more reliable/resilient/replicated (with a lower operational effort) provider of HBase tables and Kafka streams, and a bit of file storage.
moonjoAWS 7y ago
Any impacted engineers looking for a new home in the Bay Area or Seattle, feel free to reach out moonjo@amazon.com to learn more about teams with Redshift.
Communitivity 7y ago
This is sad. I remember the MapR demo at the NVidia GTC a couple years ago, and their demo was amazing. I hope they pull a rabbit out of the hat and are able to continue, but if they don't then I hope they Open Source as much as they can.
xkgt 7y ago
As many others pointed out, these product vendors are getting killed by the cloud (I am not saying whether it is good or bad). On-demand compute and storage scalability of cloud makes perfect sense for Big data infrastructure. These companies are failing because they can't establish a use case of running their product on cloud. It is ironic because when AWS started EMR, MapR was one of the distributions they offered via EMR[1]. Over period of time, they weened off MapR and stopped offering them as deployment option. So it is another case of AWS cannibalizing its own third party ecosystem. Once this was gone, MapR got reduced just another marketplace partner [2] and their additional licensing cost didn't make sense when native EMR was sufficient for most use case. [1] https://aws.amazon.com/emr/mapr/pricing/ https://aws.amazon.com/emr/mapr/pricing/ [2] https://mapr.com/partners/partner/amazon-elastic-mapreduce-and-mapr/ https://mapr.com/partners/partner/amazon-elastic-mapreduce-a...
- jcims 7y ago
  Which has to be a gut punch because EMR feels like a bag of bolts rather than any kind of polished system.
  - sitkack 7y ago
    And MapR is so much more efficient than vanilla Hadoop, AWS definitely makes more money off the less polished yet mainstream version.
    - xkgt 7y ago
      IMO it is not a fair comparison between EMR and MapR flavor of hadoop as they are designed for different usecases. MapR and other flavors of hadoop are engineered to make the task of administering a long running cluster easy. In comparison, EMR has a rudimentary set of tools for peaking into a running cluster. Whereas AWS offering is ideally suited for transient clusters. It relies on other solutions like Athena, Redshift Spectrum to cater for ad-hoc use cases such as querying and reporting. In this regard, EMR has much better support for programmability and elastic resource provisioning which is really important for transient clusters.
atombender 7y ago
Lots of interesting discussion here about billions of files and what not. I don't really work with big data -- the largest amounts of data I work with are in the range of tens of millions at most, and my modest "ETL" is about reading from APIs and cleaning up incoming data into consistent schemas, refining data (e.g. geocoding addresses, or correlating against public demographic records/statistics), a process that is super fast and doesn't even require multiple nodes for the most part -- so I never encounter anything I can't solve with some Postgres or Redis. What I'd love to know is what everyone is actually doing with their huge clusters. I can understand the need for big data in hard sciences (CERN, or genetics, or similar), and of course machine learning (e.g. image feature extraction), and then there are real time auction use cases like ad servers that probably have some big data component. What else are people doing out there?
- heyyyouu 7y ago
  This is a great question -- I'd love to know this too. I watch this space from the enterprise perspective and I see a lot of big data talk/prep but not so much execution, if that makes sense.

108 points by heyyyouu 7y ago

So, that would mean the hadoop distro market would be down to just cloudera? A single vendor market is not a market, it's a legacy product.

russtrotter 7y ago
AFAIK, AWS's EMR and Glue leverage Hadoop under the covers. Does that count towards that market? and/or at least keeping Hadoop development active?
- derefr 7y ago
  Hadoop ≠ Hadoop distro. Hadoop is fine and healthy; what is in question is whether larger repackagings of it with other vendor-specific components is an interesting product any more.
- filesystem 7y ago
  Google's dataproc and Azure's HDInsight as well. The cloud players are swallowing whatever is left of the hadoop market.
- threeseed 7y ago
  EMR is their managed distro of Hadoop. DataPipeline is their managed Hadoop for ETL. Glue ETL is their managed Spark. And it all absolutely counts towards the market.
buryat 7y ago
Everybody is moving to the cloud so EMR, Dataproc, Dataflow, etc. are still going to be around Plus everyone is moving away from Hadoop and HDFS, so it's Spark, S3, GCS, Databricks, etc
badpun 7y ago
Is Microsoft Office or Microsoft Windows a legacy product?
nwsm 7y ago
Azure's HDInsight all-in-one suite uses it.
- zjaffee 7y ago
  HDInsite is built by hortonworks which was acquired by cloudera.
- opportune 7y ago
  In addition to HDInsight being cloudera, it's also basically a legacy product. Azure technical solutions people are now all pushing databricks. And personally, when designing a new project, I don't see why you would use HDInsight instead of a bespoke distributed vm architecture with one (azure storage) or two-tier storage (one local to the vm, one in Azure storage) unless you really need to use some hadoop jar you found lying around.
jamesblonde 7y ago
Actually, there is another vendor. A new one - www.logicalclocks.com - we are selling a next-generation version of Hadoop, with consistent distributed metadata for HDFS, it's built on TLS/SSL (not kerberos), and it's laser focussed on Data Science. We have customers with racks with 10s of GPUs training models with 10s of TBs of feature data. And we also built the industry's first enterprise feature store for ML on this platform. References: https://www.logicalclocks.com/feature-store/ https://www.logicalclocks.com/feature-store/ https://www.logicalclocks.com/introducing-hops-hadoop/ https://www.logicalclocks.com/introducing-hops-hadoop/

kwillets 7y ago

MapR has been reduced. Hadoop was the floppy disk of big data. Ubiquitous, but always beat by other solutions.

threeseed 7y ago
This is simply not true. Hadoop is growing massively. All that is happening is that the vendors are being killed by the cloud.
- spork12 7y ago
  I've seen Hadoop and HDFS dying off, and being replaced by Spark, S3, and Azure Storage. There is also a lot of good work being done by DataBricks in creating an easy to manage spark platform.

kod 7y ago

Hope they're able to open source some of their tech if this does come to pass.

notyourday 7y ago

To me, and a few people that I know, the most interesting part of MapR was their NFS server over multiple nodes. It solved a real problem that real customers (i.e. enterprises with money for whom tech made stuff work and not was work in itself) were willing to pay a lot of money for ( see Isilon, EMC, NetApp, etc ) but that was a non-sexy part of the MapR business -- Big Data part was much sexier. Over the last 5 years the real money making enterprises solved their "it should look like a big file system" problem so "the rest of our stuff works" issue stopped being an issue ( in a process Isilon got bought by the EMC, EMC got bought by Dell ) either by buying those specialized solutions, moving to object store or building home grown systems that worked with their specific applications and big data people went with newer, more shiny big data solutions, leaving MapR with no market.

heyyyouu 7y ago
Which of the shinier, newer solutions do you think people have found more attractive in particular? Genuinely curious, thanks!
- notyourday 7y ago
  I'm not someone very much in the "big data" [0] but from what I see streaming analytics especially Spark-based solutions are eating the world that used to be dominated by Hadoop. I also think cloud based data lakes and tooling around them seriously decreased the appeal of Hadoop. [0] If I can fit your big data into my memory, it is not "big data".
  - brokensegue 7y ago
    i mean spark is often deployed on top of yarn where it's still in the "hadoop" sphere.
    - notyourday 7y ago
      But I'm not loading up on my SuperMicro, Dell or HP servers. So there's no need to buy a decent Hadoop implementation from MapR.

joehandzik 7y ago

We're seeing a lot of regret around sprawling Hadoop deployments, so this doesn't surprise me. Other Hadoop vendors (vendor?) pivoting to machine learning is a bandaid as the compute capabilities scale beyond HDFS's performance limitations. Look towards new-gen startups around NVME/NVMEOF (WekaIO, Excelero, E8, etc etc) to fill the void. The question is going to be: will anyone provide an intelligent way to maintain compatibility with applications that expect to interface with HDFS rather than POSIX? It's a bit of a gap right now from what we see.

joehandzik 7y ago
Note that apparently the layoffs have already happened. Engineers are already interviewing elsewhere.
capkutay 7y ago
Hadoop's whole appeal was it was cheap and scalable. Did it actually work without serious engineering teams maintaining each distribution? absolutely not. the hype was purely VC funded. fast forward 10 years and hadoop has basically been killed off by hosted storage services that are more expensive but 10000x easier to manage.
- dmix 7y ago
  I'm not really familiar with this industry. Are these Hadoop clusters hosted locally by these businesses? And MapR et all were providing the software/consulting to help manage them? Then I'm guessing better cloud hosted options came out offering similar capabilities. If that's the case was it really a big surprise that "cloud" hosting would eat any self-hosted platform's lunch?
  - capkutay 7y ago
    yes mostly locally hosted back then. the idea was you'd have this massive distributed file system to hold all your data, and VC backed comopanies like Cloudera, MapR, Hortonworks promised everyone that they'd build the SQL layer, the data warehousing, BI, and all the other enterprise features you'd need to basically replace your expensive Teradata, Oracle Exadata, and other data warehousing systems. But of course it didn't play out that way.

bitL 7y ago

Weren't they killed the moment Spark appeared, having heavily invested in accelerating Hadoop? I am surprised that Cloudera is still around...

SEJeff 7y ago
And spark was sort of killed by Storm / Heron as even Spark Streaming does microbatches and isn't truly realtime.
- Dunedan 7y ago
  Wasn't it the other way around? Wasn't Storm killed by Spark? At least once Spark was out and gained traction, nobody in my bubble talked about Storm anymore. But I guess it depends on the bubble you're in.
  - brokensegue 7y ago
    yeah their timeline seems backwards
- suresk 7y ago
  Wouldn't Flink be more likely to replace Spark? I don't hear about a lot of people using Storm these days, but I guess a lot of this depends on whatever bubble you happen to be in.
- capkutay 7y ago
  in what world was spark killed by storm/heron? last i checked the spark conference had over 10k attendees this year. storm is barely supported and heron never picked up steam.
- threeseed 7y ago
  What ? Spark Streaming is just one part of Spark and the far less used part as well. Spark hasn't been killed by anything and is still heavily used for ETL and ML.
- mruts 7y ago

purplezooey 7y ago

I think they'll be fine. At my last company we had a large MapR cluster and it was hands down more reliable and user-friendly than anything else. Maybe Cloudera will pick them up.

jrg 7y ago
We've used them as a more reliable/resilient/replicated (with a lower operational effort) provider of HBase tables and Kafka streams, and a bit of file storage.

moonjoAWS 7y ago

Any impacted engineers looking for a new home in the Bay Area or Seattle, feel free to reach out moonjo@amazon.com to learn more about teams with Redshift.

Communitivity 7y ago

This is sad. I remember the MapR demo at the NVidia GTC a couple years ago, and their demo was amazing. I hope they pull a rabbit out of the hat and are able to continue, but if they don't then I hope they Open Source as much as they can.

xkgt 7y ago

As many others pointed out, these product vendors are getting killed by the cloud (I am not saying whether it is good or bad). On-demand compute and storage scalability of cloud makes perfect sense for Big data infrastructure. These companies are failing because they can't establish a use case of running their product on cloud. It is ironic because when AWS started EMR, MapR was one of the distributions they offered via EMR[1]. Over period of time, they weened off MapR and stopped offering them as deployment option. So it is another case of AWS cannibalizing its own third party ecosystem. Once this was gone, MapR got reduced just another marketplace partner [2] and their additional licensing cost didn't make sense when native EMR was sufficient for most use case. [1] https://aws.amazon.com/emr/mapr/pricing/ https://aws.amazon.com/emr/mapr/pricing/ [2] https://mapr.com/partners/partner/amazon-elastic-mapreduce-and-mapr/ https://mapr.com/partners/partner/amazon-elastic-mapreduce-a...

jcims 7y ago
Which has to be a gut punch because EMR feels like a bag of bolts rather than any kind of polished system.
- sitkack 7y ago
  And MapR is so much more efficient than vanilla Hadoop, AWS definitely makes more money off the less polished yet mainstream version.
  - xkgt 7y ago
    IMO it is not a fair comparison between EMR and MapR flavor of hadoop as they are designed for different usecases. MapR and other flavors of hadoop are engineered to make the task of administering a long running cluster easy. In comparison, EMR has a rudimentary set of tools for peaking into a running cluster. Whereas AWS offering is ideally suited for transient clusters. It relies on other solutions like Athena, Redshift Spectrum to cater for ad-hoc use cases such as querying and reporting. In this regard, EMR has much better support for programmability and elastic resource provisioning which is really important for transient clusters.

atombender 7y ago

Lots of interesting discussion here about billions of files and what not. I don't really work with big data -- the largest amounts of data I work with are in the range of tens of millions at most, and my modest "ETL" is about reading from APIs and cleaning up incoming data into consistent schemas, refining data (e.g. geocoding addresses, or correlating against public demographic records/statistics), a process that is super fast and doesn't even require multiple nodes for the most part -- so I never encounter anything I can't solve with some Postgres or Redis. What I'd love to know is what everyone is actually doing with their huge clusters. I can understand the need for big data in hard sciences (CERN, or genetics, or similar), and of course machine learning (e.g. image feature extraction), and then there are real time auction use cases like ad servers that probably have some big data component. What else are people doing out there?

heyyyouu 7y ago
This is a great question -- I'd love to know this too. I watch this space from the enterprise perspective and I see a lot of big data talk/prep but not so much execution, if that makes sense.