5 ms·

Interesting post. Out of curiosity: aren't there others who have developed cluster storage exposed as NFS / FUSE?

notacoward 7y ago
Absolutely. In fact, that was my last project, which I worked on for almost a decade. Unfortunately, it's a tough market. At the low end, everyone thinks they can do it themselves. At the high end, you're up against either the "POSIX is dead let's use an object store" crowd or the enterprise EMC/NetApp/Isilon crowd. On the other side of the ledger, development costs are high. Developers are harder to find, more equipment is necessary, and the testing bar is higher, compared to other kinds of software. Low revenue + high cost = life sucks. That's why I work on a bespoke not-quite-filesystem now.
- noir-york 7y ago
  Thank you very much for taking the time!
stormbeard 7y ago
Nutanix and VMWare have been doing that pretty successfully since ~2012.
notyourday 7y ago
They all sucked. MapR sucked the least. Considering a comparable EMC solution was ~20x more expensive and less flexible, MapR happened to be in the right place at the right time with the right software but blew it because they decided they knew better what the customers needed to buy vs what the customers wanted to buy. Fundamentally, file system clusters are a very difficult business to get into and make enough money to justify being in that business. On a low end there's open source stuff that kind of sort of works. Selling consulting on a top of it is at best a ramen-profitable business. No one is buying a Ceph-consulting for a million dollars if they run a business that needs that size of storage -- the data is too valuable to do a semi-custom solution and be a test case for Ceph, Gluster, etc. On the top end there are EMCs of the world with $350K/node pricing + yearly 10% support contract. Finally, there's a "rewrite the app" alternative ( call it 1 year, million $ price tag to move to object store ) that a customer company is always considering. So the sweet spot is enterprise solution with a license cost of about $12k-$24k per year per node which has EMC/Isilon/NetApp-like functionality that unbundles software from hardware. The tricky part is that it needs to be a proven solution that works on the comparable EMC/Isilon/NetApp sized deployments. To do that one needs to have a lot of excellent engineers that cost a lot of money, a lot of sales engineers that intrinsically understand the commonality between the customers' requirements and can explain it to those engineers and a lot of very expensive test beds.
- noir-york 7y ago
  Thanks for taking the time! Its fascinating learning about this in an area (storage) I know nothing about.
- mbreese 7y ago
  I remember being in a pitch meeting with MapR a few years ago. It was for a large academic institution’s HPC group. We were all very interested in their NFS/HDFS tools, which would have met a very critical need. They just kept talking about the Hadoop workflows, which wouldn’t have flown at all for this group. Most HPC workflows/tools don’t map very well to Hadoop or at the minimum would have required tools to have been rewritten. We really wanted to spend the money, but as far as I remember, we didn’t end up doing anything more than a test install. I’d still love a middle option that could handle petabyte scale storage without the Isilon price tag.
  - hc91 7y ago
    CERN?
    - mbreese 7y ago
      No, this was at Stanford a few years ago, so not close to CERN level HPC. Our lab has modest compute requirements, but need a ton of storage, so we were very interested in a mid-range storage option.
- vkoepnick 7y ago
  The sweet spot you describe sounds like Quobyte to me.
  - notacoward 7y ago
    ...except for the "proven solution" part, Quobyte marketing VP.
    - vkoepnick 7y ago

Absolutely. In fact, that was my last project, which I worked on for almost a decade. Unfortunately, it's a tough market. At the low end, everyone thinks they can do it themselves. At the high end, you're up against either the "POSIX is dead let's use an object store" crowd or the enterprise EMC/NetApp/Isilon crowd. On the other side of the ledger, development costs are high. Developers are harder to find, more equipment is necessary, and the testing bar is higher, compared to other kinds of software. Low revenue + high cost = life sucks. That's why I work on a bespoke not-quite-filesystem now.

noir-york 7y ago
Thank you very much for taking the time!

stormbeard 7y ago

Nutanix and VMWare have been doing that pretty successfully since ~2012.

notyourday 7y ago

They all sucked. MapR sucked the least. Considering a comparable EMC solution was ~20x more expensive and less flexible, MapR happened to be in the right place at the right time with the right software but blew it because they decided they knew better what the customers needed to buy vs what the customers wanted to buy. Fundamentally, file system clusters are a very difficult business to get into and make enough money to justify being in that business. On a low end there's open source stuff that kind of sort of works. Selling consulting on a top of it is at best a ramen-profitable business. No one is buying a Ceph-consulting for a million dollars if they run a business that needs that size of storage -- the data is too valuable to do a semi-custom solution and be a test case for Ceph, Gluster, etc. On the top end there are EMCs of the world with $350K/node pricing + yearly 10% support contract. Finally, there's a "rewrite the app" alternative ( call it 1 year, million $ price tag to move to object store ) that a customer company is always considering. So the sweet spot is enterprise solution with a license cost of about $12k-$24k per year per node which has EMC/Isilon/NetApp-like functionality that unbundles software from hardware. The tricky part is that it needs to be a proven solution that works on the comparable EMC/Isilon/NetApp sized deployments. To do that one needs to have a lot of excellent engineers that cost a lot of money, a lot of sales engineers that intrinsically understand the commonality between the customers' requirements and can explain it to those engineers and a lot of very expensive test beds.

noir-york 7y ago
Thanks for taking the time! Its fascinating learning about this in an area (storage) I know nothing about.
mbreese 7y ago
I remember being in a pitch meeting with MapR a few years ago. It was for a large academic institution’s HPC group. We were all very interested in their NFS/HDFS tools, which would have met a very critical need. They just kept talking about the Hadoop workflows, which wouldn’t have flown at all for this group. Most HPC workflows/tools don’t map very well to Hadoop or at the minimum would have required tools to have been rewritten. We really wanted to spend the money, but as far as I remember, we didn’t end up doing anything more than a test install. I’d still love a middle option that could handle petabyte scale storage without the Isilon price tag.
- hc91 7y ago
  CERN?
  - mbreese 7y ago
    No, this was at Stanford a few years ago, so not close to CERN level HPC. Our lab has modest compute requirements, but need a ton of storage, so we were very interested in a mid-range storage option.
vkoepnick 7y ago
The sweet spot you describe sounds like Quobyte to me.
- notacoward 7y ago
  ...except for the "proven solution" part, Quobyte marketing VP.
  - vkoepnick 7y ago