5 ms·

Good move for both companies. The surplus of 'enterprise hadoop' companies was created by a mixture of hype and a peak in VC investment in open source. The fac

Good move for both companies. The surplus of 'enterprise hadoop' companies was created by a mixture of hype and a peak in VC investment in open source. The fact that Cloudera, Hortonworks, MapR were all founded and raised $100m+ around the same time was a bit superfluous for the whole market. Hard to say where this leaves MapR now. They seem to be the odd man out in terms of growth and adoption.

jamesblonde 8y ago
We have a startup, Logical Clocks (www.logicalclocks.com), that just raised money to sell our data-science next-generation Hadoop stack. It has a new verison of HDFS called HopsFS (Nvme storage, distributed metadata) and support for GPUs in YARN. And distributed tensorflow, Spark, Airflow, Flink. So, what does that mean for us? I am shellshocked. I expect prices to increase (good for us). What else should we expect?
- agibsonccc 8y ago
  Both Horton and Cloudera require a huge number of partner resellers and consultants. What horton and cloudera have isn't just software, but brand, an install base, and the ability to back it up. They merged partially because they were both being cannibalized by services revenue they couldn't get rid of. Now they are struggling to move to the cloud (see Atlas now) I'm not sure yet another hadoop distro with a bunch of 1 off tooling that is supposedly faster is the answer. Why is all that stuff even needed?
  - jamesblonde 8y ago
    Adam, what about Java for deep learning? Is that needed?
    - agibsonccc 8y ago
      According to my customers and users yes? Granted, we do a lot more than just that though. Dl4j itself has a decent sized user base. Ranging likely from your phone maker to your bank and retail store. We have our own software distro too which is why I'm commenting on this. We don't try to boil the ocean with a bunch of tech though. There's a whole new crop of companies focusing on solving bits of the ML problem well rather than trying to do storage and god knows what else. My point here about you guys is you're trying to compete in what is largely a commodity market. People don't need all this stuff. Simplicity won here. It's not about better tech. You guys have the same pitch MapR does and largely the same problem: Better tech is only part of the problem with adoption. You need customers, users, and a clear business model when going to market. Cloudera and Horton ran one playbook that at least somewhat worked (it got them public) and now they can focus on competing with the cloud vendors, which made the right decision and just made commonly used software easy to use.
      - jamesblonde 8y ago
        We don't. We are the only on-premise vendor with proper support for GPUs and python. We are a full data science stack, backed by Hadoop. We even have Kubernetes for model serving. And python in the cluster (with conda environments). And we have customers and funding. Nothing like MapR. And none of the legacy mapreduce crap.
        agibsonccc 8y ago
        You bundle way more than they do and on top of that have your own file system just like mapr does. Your pitch is still about differentiated tech, not a large install base, a differentiated business model and something related to people like a good partner ecosystem. Your pitch here requires tons of services. People don't know how to use all of this stuff especially on prem. It takes more than just code to build a business. I say this as someone who's been doing this since 2013. It's not easy.
        jamesblonde 8y ago
        Ok, now you're changing your angle. Differentiated tech is what we are all about - that is ok by me (for now). If you want to train DNNs on a hundred GPUs today on-premise on TensorFlow, come to us, we can do it. They can't.
        agibsonccc 8y ago
        I don't think I changed my angle here? I'm still addressing the same point. Tech doesn't matter. Simplicity does. Even in our own product line, we only do a small subset of this. We don't even require a cluster to run. We also work with tech that people use. You are currently competing with horovod and kubeflow. eg: "competing with free" You need more than that to survive. Generally, that comes down to services.
        ironchef 8y ago
        A slightly different wording is it comes to using the tech... which is either through services or through adoption / operationalization (which... with complexity... is very difficult)
- SpaceManNabs 8y ago
  You support Flink too? Nice! I am glad to see more and more people eyeing it as a viable framework for streaming. Do you support Kubernetes too?
  - jamesblonde 8y ago
    Yes, but for model serving. Not for Flink or Spark. That's on YARN, for now.
- HenryR 8y ago
  I am surprised that anyone is trying to differentiate on storage at this time, precisely when that's the part of the stack that's being cannibalized by the cloud vendors (look at the rate of innovation in HDFS over time; the effort is going elsewhere). Are you just targeting on-premise clusters, or is there some differentiation planned for the cloud as well?
  - jamesblonde 8y ago
    We think that there is a niche for a higher performance dist FS than S3. We have integrated NVMe hardware with our HDFS implementation (HopsFS) and made its metadata layer distributed. NVMe means you can, for example, work with datasets with millions of files for deep learning - instead of having to munge them into parquet files because your FS is slowing down your machine learning pipeline. Reference: https://www.logicalclocks.com/millions-and-millions-of-files-deep-learning-at-scale-with-hopsfs/ https://www.logicalclocks.com/millions-and-millions-of-files... We have also redesigned the stack around our distributed metadata layer. We are primarily targeting on-prem right now, but HopsFS would be the fastest DFS in the cloud if you ran it there today.
    - Game_Ender 8y ago
      How does HopsFS compare to Lustre?
purplezooey 8y ago
My current company uses MapR and I have to say their software is impressive. It's much easier to use than CDH or HDP. I think they'll do fine.
dirkgently 8y ago
> surplus Define surplus. Why having 3 or 4 companies in 1B plus market is surplus?
- dfee 8y ago
  Because there are typically uneven returns due to the winner-takes-most style of internet businesses. Differentiation is of course key, then.
  - PanosJee 8y ago
    This is not an Internet business (no network effects), switching is not that hard on OSS stacks
batoure 8y ago
Don’t count MapR down, their commitment to developing ports and tools for the ecosystem outside the jvm was very forward thinking. Their internal engineering talent either makes them a prime aquaition target for CDH in future or a fierce competitor as systems like kudu become more common and production ready.
- tumanian 8y ago
  MapR is great while you are in their ecosystem - once you step outside and try to bring in technology thats not supported by them, you run into little showstopper bugs and complete lack of documentation and community support.
  - purplezooey 8y ago
    Our experience has not been consistent with that. For example we use MapR-DB and have never had a single issue with HBase compatibility or bugs. My DS team loves the database especially it's quite good.

jamesblonde 8y ago

We have a startup, Logical Clocks (www.logicalclocks.com), that just raised money to sell our data-science next-generation Hadoop stack. It has a new verison of HDFS called HopsFS (Nvme storage, distributed metadata) and support for GPUs in YARN. And distributed tensorflow, Spark, Airflow, Flink. So, what does that mean for us? I am shellshocked. I expect prices to increase (good for us). What else should we expect?

agibsonccc 8y ago
Both Horton and Cloudera require a huge number of partner resellers and consultants. What horton and cloudera have isn't just software, but brand, an install base, and the ability to back it up. They merged partially because they were both being cannibalized by services revenue they couldn't get rid of. Now they are struggling to move to the cloud (see Atlas now) I'm not sure yet another hadoop distro with a bunch of 1 off tooling that is supposedly faster is the answer. Why is all that stuff even needed?
- jamesblonde 8y ago
  Adam, what about Java for deep learning? Is that needed?
  - agibsonccc 8y ago
    According to my customers and users yes? Granted, we do a lot more than just that though. Dl4j itself has a decent sized user base. Ranging likely from your phone maker to your bank and retail store. We have our own software distro too which is why I'm commenting on this. We don't try to boil the ocean with a bunch of tech though. There's a whole new crop of companies focusing on solving bits of the ML problem well rather than trying to do storage and god knows what else. My point here about you guys is you're trying to compete in what is largely a commodity market. People don't need all this stuff. Simplicity won here. It's not about better tech. You guys have the same pitch MapR does and largely the same problem: Better tech is only part of the problem with adoption. You need customers, users, and a clear business model when going to market. Cloudera and Horton ran one playbook that at least somewhat worked (it got them public) and now they can focus on competing with the cloud vendors, which made the right decision and just made commonly used software easy to use.
    - jamesblonde 8y ago
      We don't. We are the only on-premise vendor with proper support for GPUs and python. We are a full data science stack, backed by Hadoop. We even have Kubernetes for model serving. And python in the cluster (with conda environments). And we have customers and funding. Nothing like MapR. And none of the legacy mapreduce crap.
      - agibsonccc 8y ago
        You bundle way more than they do and on top of that have your own file system just like mapr does. Your pitch is still about differentiated tech, not a large install base, a differentiated business model and something related to people like a good partner ecosystem. Your pitch here requires tons of services. People don't know how to use all of this stuff especially on prem. It takes more than just code to build a business. I say this as someone who's been doing this since 2013. It's not easy.
        jamesblonde 8y ago
        Ok, now you're changing your angle. Differentiated tech is what we are all about - that is ok by me (for now). If you want to train DNNs on a hundred GPUs today on-premise on TensorFlow, come to us, we can do it. They can't.
        agibsonccc 8y ago
        I don't think I changed my angle here? I'm still addressing the same point. Tech doesn't matter. Simplicity does. Even in our own product line, we only do a small subset of this. We don't even require a cluster to run. We also work with tech that people use. You are currently competing with horovod and kubeflow. eg: "competing with free" You need more than that to survive. Generally, that comes down to services.
        ironchef 8y ago
        A slightly different wording is it comes to using the tech... which is either through services or through adoption / operationalization (which... with complexity... is very difficult)
SpaceManNabs 8y ago
You support Flink too? Nice! I am glad to see more and more people eyeing it as a viable framework for streaming. Do you support Kubernetes too?
- jamesblonde 8y ago
  Yes, but for model serving. Not for Flink or Spark. That's on YARN, for now.
HenryR 8y ago
I am surprised that anyone is trying to differentiate on storage at this time, precisely when that's the part of the stack that's being cannibalized by the cloud vendors (look at the rate of innovation in HDFS over time; the effort is going elsewhere). Are you just targeting on-premise clusters, or is there some differentiation planned for the cloud as well?
- jamesblonde 8y ago
  We think that there is a niche for a higher performance dist FS than S3. We have integrated NVMe hardware with our HDFS implementation (HopsFS) and made its metadata layer distributed. NVMe means you can, for example, work with datasets with millions of files for deep learning - instead of having to munge them into parquet files because your FS is slowing down your machine learning pipeline. Reference: https://www.logicalclocks.com/millions-and-millions-of-files-deep-learning-at-scale-with-hopsfs/ https://www.logicalclocks.com/millions-and-millions-of-files... We have also redesigned the stack around our distributed metadata layer. We are primarily targeting on-prem right now, but HopsFS would be the fastest DFS in the cloud if you ran it there today.
  - Game_Ender 8y ago
    How does HopsFS compare to Lustre?

purplezooey 8y ago

My current company uses MapR and I have to say their software is impressive. It's much easier to use than CDH or HDP. I think they'll do fine.

dirkgently 8y ago

> surplus Define surplus. Why having 3 or 4 companies in 1B plus market is surplus?

dfee 8y ago
Because there are typically uneven returns due to the winner-takes-most style of internet businesses. Differentiation is of course key, then.
- PanosJee 8y ago
  This is not an Internet business (no network effects), switching is not that hard on OSS stacks

batoure 8y ago

Don’t count MapR down, their commitment to developing ports and tools for the ecosystem outside the jvm was very forward thinking. Their internal engineering talent either makes them a prime aquaition target for CDH in future or a fierce competitor as systems like kudu become more common and production ready.

tumanian 8y ago
MapR is great while you are in their ecosystem - once you step outside and try to bring in technology thats not supported by them, you run into little showstopper bugs and complete lack of documentation and community support.
- purplezooey 8y ago
  Our experience has not been consistent with that. For example we use MapR-DB and have never had a single issue with HBase compatibility or bugs. My DS team loves the database especially it's quite good.