4 ms·

I can't speak for the quality of other object storage providers, but being in the storage business I can say that if someone is running Ceph, find another provi

I can't speak for the quality of other object storage providers, but being in the storage business I can say that if someone is running Ceph, find another provider.

kawera 9y ago
Care to elaborate a little?
scurvy 9y ago
This is pure conjecture and FUD. Lemme guess, you work at EMC? I can refute your comment by saying that we have never lost a bit of data while running Ceph. I can't say the same for EMC tech (Isilon).
- nik736 9y ago
  A lot of companies using Ceph at scale are facing huge issues (OVH, etc.), so he is not wrong. Why take the risk of going with a solution that is known to cause issues? Has nothing to do with EMC.
  - true_religion 9y ago
    I believe OVH uses openstack and not ceph.
    - nik736 9y ago
      You can use Ceph together with OpenStack. They used ceph for their cloud services but had huge problems. If I am not mistaken they completely threw out Ceph by now. http://travaux.ovh.net/?do=details&id=20636 http://travaux.ovh.net/?do=details&id=20636 http://status.ovh.net/?do=details&id=14139 http://status.ovh.net/?do=details&id=14139 http://travaux.ovh.com/?do=details&id=20490 http://travaux.ovh.com/?do=details&id=20490 http://travaux.ovh.net/?do=details&id=26382 http://travaux.ovh.net/?do=details&id=26382 and so on... They took months to figure out what's going on.
      - mattbillenstein 9y ago
        Any idea what the underlying issues with Ceph were? My story is a bit dated, but we went from gluster to ceph to moosefs at one startup. Gluster had odd performance problems (slow metadata operations - scatter/gather rpcs and whatnot I would guess) and it was hard to know from the logs what was going on. Ceph was very very early at this point, but part of it ran as a kernel module and the first time it oops'd, I deleted that with fire. MooseFS ran all in userspace, had good tools for observability into the state of the cluster, and the source code was simple and clean. It didn't have a good story around multi-master at that time, but I think that is improved now.
        zzzcpan 9y ago
        I can't find now, was MooseFS the one using MySQL for metadata?
        daviesliu 9y ago
        MooseFS have a highly optimized service to load all the metadata into memory (similar to Redis).
        nik736 9y ago
        No clue what the underlying issue was but when reading: "We have about 200 harddisk in this cluster... 1 of the disks was broken and we removed it. For some reasons, Ceph stopped to working : 17 objectfs are missed. It should not." I think the underlying issue is simply "Ceph" ;-)
        scurvy 9y ago
        User error
        mattbillenstein 9y ago
        zzzcpan - no, metadata was in RAM which made things like directory listings and whatnot that were very slow in gluster very fast in MooseFS
  - pinewurst 9y ago
    Thanks for that! I've talked to a lot of large-ish commercial Ceph customers and they seem to spend a lot of time building kludge-arounds for support. And tend to live terrified that the whole clumsy edifice will come crashing down at the cost of their jobs. Also too, Ceph is block, object and file. Block is ok up to a point, object is dubious and file is utterly untrustworthy. At least at any kind of real scale - 3 servers in a rack aren't "scale". Why must someone who isn't a Ceph fan (and I fail to see why storage systems are a "fan" activity) live in the evil pockets of EMC? I know people who've smoked for years and don't have any sign of lung cancer either.
    - scurvy 9y ago
      Ceph is all K/V objects underneath. To say that "block is OK" but "object is dubious"...is silly. Care to share what "kludge-arounds" there are for support? Red Hat offers commercial support in case you need to phone a friend.
  - scurvy 9y ago
    OVH isn't exactly a shining example of a quality engineering organization. Simple web searches show how they have misused things and cause large outages. Ceph is very reliable and durable. We've actually gone out of our way to try and corrupt data, but we failed every time. It always repaired the data correctly and brought things back into a good working state. Ceph and Yahoo run very large Ceph clusters at scale, too.
- dang 9y ago
  > Lemme guess, you work at This crosses into personal attack. That's not ok here, so please don't. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
dstroot 9y ago
Who is running what?
polskibus 9y ago
What's a better open source alternative to Ceph?
- AFNobody 9y ago
  There isn't a better open source alternative to Ceph. However, improperly designed/architected you will end up with serious scaling issues.
  - scurvy 9y ago
    Testing testing testing! It's important to actually test things.
    - AFNobody 9y ago
      Yes it is and not just initial testing but at scale with full volume. ;)
- mattbillenstein 9y ago
  MooseFS
- halbritt 9y ago
  The issue with Ceph isn't that it's some how deficient. It's amazing. The problem is that it's difficult to engineer correctly and hard to troubleshoot. There is no other software, open source or otherwise that works quite as well as Ceph for providing durability and scale. ScaleIO gets high marks for block storage performance compared to Ceph. It's not quite as durable and lacks some other features, but people seem to like it.
  - tatersolid 9y ago
    > The problem is that it's difficult to engineer correctly and hard to troubleshoot. These sound like... problems with Ceph. Sane deployment, management, and troubleshooting are critical features of any distributed system. If they’re not there the system isn’t “ready”.
zzzcpan 9y ago
If you are relying on a single object storage provider and cannot survive downtime, data loss or simply being very slow at times you will never find a good one. Expect things to fail. Distributed systems are not that trivial for a random object storage provider to have enough expertise to run Ceph or any other open source solution at scale with no issues.
- scurvy 9y ago
  Or you should purchase commercial support for said storage... Salesforce runs several large Ceph clusters, and they have a dedicated team to run it. If you can't invest in the employees, you should invest in commercial support. Salesforce also commits a lot of updates and patches back to the Ceph community

I can't speak for the quality of other object storage providers, but being in the storage business I can say that if someone is running Ceph, find another provider.

kawera 9y ago

Care to elaborate a little?

scurvy 9y ago

This is pure conjecture and FUD. Lemme guess, you work at EMC? I can refute your comment by saying that we have never lost a bit of data while running Ceph. I can't say the same for EMC tech (Isilon).

nik736 9y ago
A lot of companies using Ceph at scale are facing huge issues (OVH, etc.), so he is not wrong. Why take the risk of going with a solution that is known to cause issues? Has nothing to do with EMC.
- true_religion 9y ago
  I believe OVH uses openstack and not ceph.
  - nik736 9y ago
    You can use Ceph together with OpenStack. They used ceph for their cloud services but had huge problems. If I am not mistaken they completely threw out Ceph by now. http://travaux.ovh.net/?do=details&id=20636 http://travaux.ovh.net/?do=details&id=20636 http://status.ovh.net/?do=details&id=14139 http://status.ovh.net/?do=details&id=14139 http://travaux.ovh.com/?do=details&id=20490 http://travaux.ovh.com/?do=details&id=20490 http://travaux.ovh.net/?do=details&id=26382 http://travaux.ovh.net/?do=details&id=26382 and so on... They took months to figure out what's going on.
    - mattbillenstein 9y ago
      Any idea what the underlying issues with Ceph were? My story is a bit dated, but we went from gluster to ceph to moosefs at one startup. Gluster had odd performance problems (slow metadata operations - scatter/gather rpcs and whatnot I would guess) and it was hard to know from the logs what was going on. Ceph was very very early at this point, but part of it ran as a kernel module and the first time it oops'd, I deleted that with fire. MooseFS ran all in userspace, had good tools for observability into the state of the cluster, and the source code was simple and clean. It didn't have a good story around multi-master at that time, but I think that is improved now.
      - zzzcpan 9y ago
        I can't find now, was MooseFS the one using MySQL for metadata?
        daviesliu 9y ago
        MooseFS have a highly optimized service to load all the metadata into memory (similar to Redis).
      - nik736 9y ago
        No clue what the underlying issue was but when reading: "We have about 200 harddisk in this cluster... 1 of the disks was broken and we removed it. For some reasons, Ceph stopped to working : 17 objectfs are missed. It should not." I think the underlying issue is simply "Ceph" ;-)
        scurvy 9y ago
        User error
      - mattbillenstein 9y ago
        zzzcpan - no, metadata was in RAM which made things like directory listings and whatnot that were very slow in gluster very fast in MooseFS
- pinewurst 9y ago
  Thanks for that! I've talked to a lot of large-ish commercial Ceph customers and they seem to spend a lot of time building kludge-arounds for support. And tend to live terrified that the whole clumsy edifice will come crashing down at the cost of their jobs. Also too, Ceph is block, object and file. Block is ok up to a point, object is dubious and file is utterly untrustworthy. At least at any kind of real scale - 3 servers in a rack aren't "scale". Why must someone who isn't a Ceph fan (and I fail to see why storage systems are a "fan" activity) live in the evil pockets of EMC? I know people who've smoked for years and don't have any sign of lung cancer either.
  - scurvy 9y ago
    Ceph is all K/V objects underneath. To say that "block is OK" but "object is dubious"...is silly. Care to share what "kludge-arounds" there are for support? Red Hat offers commercial support in case you need to phone a friend.
- scurvy 9y ago
  OVH isn't exactly a shining example of a quality engineering organization. Simple web searches show how they have misused things and cause large outages. Ceph is very reliable and durable. We've actually gone out of our way to try and corrupt data, but we failed every time. It always repaired the data correctly and brought things back into a good working state. Ceph and Yahoo run very large Ceph clusters at scale, too.
dang 9y ago
> Lemme guess, you work at This crosses into personal attack. That's not ok here, so please don't. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

dstroot 9y ago

Who is running what?

polskibus 9y ago

What's a better open source alternative to Ceph?

AFNobody 9y ago
There isn't a better open source alternative to Ceph. However, improperly designed/architected you will end up with serious scaling issues.
- scurvy 9y ago
  Testing testing testing! It's important to actually test things.
  - AFNobody 9y ago
    Yes it is and not just initial testing but at scale with full volume. ;)
mattbillenstein 9y ago
MooseFS
halbritt 9y ago
The issue with Ceph isn't that it's some how deficient. It's amazing. The problem is that it's difficult to engineer correctly and hard to troubleshoot. There is no other software, open source or otherwise that works quite as well as Ceph for providing durability and scale. ScaleIO gets high marks for block storage performance compared to Ceph. It's not quite as durable and lacks some other features, but people seem to like it.
- tatersolid 9y ago
  > The problem is that it's difficult to engineer correctly and hard to troubleshoot. These sound like... problems with Ceph. Sane deployment, management, and troubleshooting are critical features of any distributed system. If they’re not there the system isn’t “ready”.

zzzcpan 9y ago

If you are relying on a single object storage provider and cannot survive downtime, data loss or simply being very slow at times you will never find a good one. Expect things to fail. Distributed systems are not that trivial for a random object storage provider to have enough expertise to run Ceph or any other open source solution at scale with no issues.

scurvy 9y ago
Or you should purchase commercial support for said storage... Salesforce runs several large Ceph clusters, and they have a dedicated team to run it. If you can't invest in the employees, you should invest in commercial support. Salesforce also commits a lot of updates and patches back to the Ceph community