4 ms·

You can use Ceph together with OpenStack. They used ceph for their cloud services but had huge problems. If I am not mistaken they completely threw out Ceph by

by nik736 9y ago

You can use Ceph together with OpenStack. They used ceph for their cloud services but had huge problems. If I am not mistaken they completely threw out Ceph by now. http://travaux.ovh.net/?do=details&id=20636 http://travaux.ovh.net/?do=details&id=20636 http://status.ovh.net/?do=details&id=14139 http://status.ovh.net/?do=details&id=14139 http://travaux.ovh.com/?do=details&id=20490 http://travaux.ovh.com/?do=details&id=20490 http://travaux.ovh.net/?do=details&id=26382 http://travaux.ovh.net/?do=details&id=26382 and so on... They took months to figure out what's going on.

mattbillenstein 9y ago
Any idea what the underlying issues with Ceph were? My story is a bit dated, but we went from gluster to ceph to moosefs at one startup. Gluster had odd performance problems (slow metadata operations - scatter/gather rpcs and whatnot I would guess) and it was hard to know from the logs what was going on. Ceph was very very early at this point, but part of it ran as a kernel module and the first time it oops'd, I deleted that with fire. MooseFS ran all in userspace, had good tools for observability into the state of the cluster, and the source code was simple and clean. It didn't have a good story around multi-master at that time, but I think that is improved now.
- zzzcpan 9y ago
  I can't find now, was MooseFS the one using MySQL for metadata?
  - daviesliu 9y ago
    MooseFS have a highly optimized service to load all the metadata into memory (similar to Redis).
- nik736 9y ago
  No clue what the underlying issue was but when reading: "We have about 200 harddisk in this cluster... 1 of the disks was broken and we removed it. For some reasons, Ceph stopped to working : 17 objectfs are missed. It should not." I think the underlying issue is simply "Ceph" ;-)
  - scurvy 9y ago
    User error
- mattbillenstein 9y ago
  zzzcpan - no, metadata was in RAM which made things like directory listings and whatnot that were very slow in gluster very fast in MooseFS
- halbritt 9y ago
  Ceph is extraordinarily complicated to run correctly. The docs aren't great and commercial support is pretty mediocre. It's an amazing piece of software, but takes a great deal of engineering to get right. Most folks won't invest that much engineering into their storage. This is why Providers like EMC and NetApp can extract 10x the cost of the raw storage from enterprises.
  - scurvy 9y ago
    The RedHat ceph docs are great and open to everyone for free. The RedHat commercial support has been pretty good for us. We presented them with 2 bugs, and they addressed both. One took a few weeks but one only took a few hours to get a hotfix started. EMC storage is absolute trash post Dell merger. Pure 100% dumpsterfire. Their customers know their systems better than they do. It's pathetic.

by nik736 9y ago

mattbillenstein 9y ago

Any idea what the underlying issues with Ceph were? My story is a bit dated, but we went from gluster to ceph to moosefs at one startup. Gluster had odd performance problems (slow metadata operations - scatter/gather rpcs and whatnot I would guess) and it was hard to know from the logs what was going on. Ceph was very very early at this point, but part of it ran as a kernel module and the first time it oops'd, I deleted that with fire. MooseFS ran all in userspace, had good tools for observability into the state of the cluster, and the source code was simple and clean. It didn't have a good story around multi-master at that time, but I think that is improved now.

zzzcpan 9y ago
I can't find now, was MooseFS the one using MySQL for metadata?
- daviesliu 9y ago
  MooseFS have a highly optimized service to load all the metadata into memory (similar to Redis).
nik736 9y ago
No clue what the underlying issue was but when reading: "We have about 200 harddisk in this cluster... 1 of the disks was broken and we removed it. For some reasons, Ceph stopped to working : 17 objectfs are missed. It should not." I think the underlying issue is simply "Ceph" ;-)
- scurvy 9y ago
  User error
mattbillenstein 9y ago
zzzcpan - no, metadata was in RAM which made things like directory listings and whatnot that were very slow in gluster very fast in MooseFS
halbritt 9y ago
Ceph is extraordinarily complicated to run correctly. The docs aren't great and commercial support is pretty mediocre. It's an amazing piece of software, but takes a great deal of engineering to get right. Most folks won't invest that much engineering into their storage. This is why Providers like EMC and NetApp can extract 10x the cost of the raw storage from enterprises.
- scurvy 9y ago
  The RedHat ceph docs are great and open to everyone for free. The RedHat commercial support has been pretty good for us. We presented them with 2 bugs, and they addressed both. One took a few weeks but one only took a few hours to get a hotfix started. EMC storage is absolute trash post Dell merger. Pure 100% dumpsterfire. Their customers know their systems better than they do. It's pathetic.