4 ms·

Anybody got experience comparing ceph vs gluster vs lustre, etc? I am interested in simplicity and resilience in case of node outage. Is any of these close to s

Anybody got experience comparing ceph vs gluster vs lustre, etc? I am interested in simplicity and resilience in case of node outage. Is any of these close to set and forget?

leonardp 6y ago
i only have experience with ceph and would argue that if you set up a cluster with version 15.x right now it is pretty close to set and forget.
Jouvence 6y ago
Lustre is not aligned at all with your requirements, so forget that one. Ceph is much more complex than Gluster, but also more capable. Honestly unless you are dealing with hundreds of TB of storage (and therefore need multiple servers anyway), I expect the complexity any distributed file-system adds is going to be detrimental to uptime and stability more often than it provides extra resilience. Use a single box with ZFS if you can, and add Gluster on top only if it can't be avoided.
- fulafel 6y ago
  And even if you need hundreds of TB of storage, in many cases it might be better to just shard over normal storage and use your backups in case of failures. Current disks are big so you don't need too many for eg 200 TB.
ibotty 6y ago
I have mildly bad experience with gluster. I did never lose data, but the maintenance load was very noticable! Ceph will rebalance by itself when adding (or removing) disks. That alone reduces maintenance by a huge amount. For Storage clusters I don't care about as much, it's also nice when a disk fails that ceph will notice and react automatically. Replacing the disk is then something you ought to to at one time, but you don't have to do pretty soon. I am running some rook-managed ceph clusters now that make even ceph infrastructure nodes automatically recoverable. But I did not test that enough yet.
notacoward 6y ago
I was a Gluster developer for about ten years, including time as a project maintainer and as part of the team responsible for the largest deployment in the world. I also worked professionally with Lustre for a couple of years, and have some familiarity with Ceph. Ceph and Lustre are both largely "set and forget" for the object-storage nodes which are the most numerous type. On the other hand, both have separate metadata servers which can be much more troublesome if/when one fails. Gluster only has one kind of node. One of the main issues here is: when do you actually give up on a node and start regenerating its data somewhere else? Starting that process and then having the node actually come back can be anywhere from a waste of time to a total disaster, so you don't want to be too "twitchy" about it, but you also don't want to run in a degraded state forever. In Gluster at least, the approach was to assume that a node's coming back unless/until explicitly told otherwise (possibly by external automation). The larger point IMO is that no distributed storage system is "set and forget" at any significant scale. At the very least you'll want robust monitoring (which to some extent has to be built into the code) and alerting, and somebody to respond to the more serious alerts. I'd give a nod to Ceph in this respect. Even better is to have your own site-specific automation for common tasks like capacity addition and upgrades. Even if the storage system itself is doing everything "right" it can get pretty messy if that's not happening in sync with other systems such as provisioning and service discovery - not to mention the systems actually using the storage. Also, even though the "POSIX is dead" folks are my sworn enemies, it's still true that an ever-decreasing number of workloads actually require an actual filesystem and its associated complexity. Not zero, probably never zero, but smaller every day. If you reasonably can get away with deploying a simpler kind of storage, I recommend it.

Anybody got experience comparing ceph vs gluster vs lustre, etc? I am interested in simplicity and resilience in case of node outage. Is any of these close to set and forget?

leonardp 6y ago

i only have experience with ceph and would argue that if you set up a cluster with version 15.x right now it is pretty close to set and forget.

Jouvence 6y ago

Lustre is not aligned at all with your requirements, so forget that one. Ceph is much more complex than Gluster, but also more capable. Honestly unless you are dealing with hundreds of TB of storage (and therefore need multiple servers anyway), I expect the complexity any distributed file-system adds is going to be detrimental to uptime and stability more often than it provides extra resilience. Use a single box with ZFS if you can, and add Gluster on top only if it can't be avoided.

fulafel 6y ago
And even if you need hundreds of TB of storage, in many cases it might be better to just shard over normal storage and use your backups in case of failures. Current disks are big so you don't need too many for eg 200 TB.

ibotty 6y ago

I have mildly bad experience with gluster. I did never lose data, but the maintenance load was very noticable! Ceph will rebalance by itself when adding (or removing) disks. That alone reduces maintenance by a huge amount. For Storage clusters I don't care about as much, it's also nice when a disk fails that ceph will notice and react automatically. Replacing the disk is then something you ought to to at one time, but you don't have to do pretty soon. I am running some rook-managed ceph clusters now that make even ceph infrastructure nodes automatically recoverable. But I did not test that enough yet.

notacoward 6y ago

I was a Gluster developer for about ten years, including time as a project maintainer and as part of the team responsible for the largest deployment in the world. I also worked professionally with Lustre for a couple of years, and have some familiarity with Ceph. Ceph and Lustre are both largely "set and forget" for the object-storage nodes which are the most numerous type. On the other hand, both have separate metadata servers which can be much more troublesome if/when one fails. Gluster only has one kind of node. One of the main issues here is: when do you actually give up on a node and start regenerating its data somewhere else? Starting that process and then having the node actually come back can be anywhere from a waste of time to a total disaster, so you don't want to be too "twitchy" about it, but you also don't want to run in a degraded state forever. In Gluster at least, the approach was to assume that a node's coming back unless/until explicitly told otherwise (possibly by external automation). The larger point IMO is that no distributed storage system is "set and forget" at any significant scale. At the very least you'll want robust monitoring (which to some extent has to be built into the code) and alerting, and somebody to respond to the more serious alerts. I'd give a nod to Ceph in this respect. Even better is to have your own site-specific automation for common tasks like capacity addition and upgrades. Even if the storage system itself is doing everything "right" it can get pretty messy if that's not happening in sync with other systems such as provisioning and service discovery - not to mention the systems actually using the storage. Also, even though the "POSIX is dead" folks are my sworn enemies, it's still true that an ever-decreasing number of workloads actually require an actual filesystem and its associated complexity. Not zero, probably never zero, but smaller every day. If you reasonably can get away with deploying a simpler kind of storage, I recommend it.