5 ms·

I'm not currently using any but I've tried: - Ceph: Very flexible. Supports many different kinds of replication. Has high overhead compared to local disk (on t

by Veratyr 10y ago

I'm not currently using any but I've tried: - Ceph: Very flexible. Supports many different kinds of replication. Has high overhead compared to local disk (on the order of ~50%) and was (for me) prone to hard to diagnose issues. Can be annoying to setup if you're not doing it on a supported Linux distro with ceph-deploy. It looks like Bluestore (a new on-disk format for data) will significantly improve performance but Bluestore is extremely RAM hungry. - GlusterFS: Much faster than Ceph but less flexible. Has odd requirements about "bricks" being the same size. Much less RAM hungry than Ceph. - A bunch of smaller ones I can't recall. Mostly discarded because they performed badly or lacked replication options (I really wanted erasure coding). In the end I'm simply sharding my data manually. It's not as scalable but it's much more performant.

_jezell_ 10y ago
For what it's worth, same here. It's easier to shard files than it is to admin these systems.
mattbillenstein 10y ago
Take a look at MooseFS -- it's been several years since I eval'd all these options and MooseFS was the best -- it is a very sensible system. edit - this is distributed block storage -- if you just need object storage, perhaps something else is in order.
- Veratyr 10y ago
  From memory MooseFS was one of the ones that lacked erasure coding. The documentation also left a lot to be desired.
- throw2016 10y ago
  I think Moosefs is file not block, like Gluster. I actually found it better than Gluster in terms of robustness and performance. It's got support for multiple masters, failover and a nice dashboard. One of those projects that should be much more well known than it is given there are few open source distributed storage solutions. The MFS devs are good but maybe lack the marketing savvy or perhaps just happy where they are.
  - mattbillenstein 10y ago
    I also had a lot of issues with Gluster -- when it was broken, it was hard to know what was wrong. With MooseFS, the dashboard would usually give you a good idea, and the logs were pretty good at showing how it was replicating data after a node or disk failed.

by Veratyr 10y ago

_jezell_ 10y ago

For what it's worth, same here. It's easier to shard files than it is to admin these systems.

mattbillenstein 10y ago

Take a look at MooseFS -- it's been several years since I eval'd all these options and MooseFS was the best -- it is a very sensible system. edit - this is distributed block storage -- if you just need object storage, perhaps something else is in order.

Veratyr 10y ago
From memory MooseFS was one of the ones that lacked erasure coding. The documentation also left a lot to be desired.
throw2016 10y ago
I think Moosefs is file not block, like Gluster. I actually found it better than Gluster in terms of robustness and performance. It's got support for multiple masters, failover and a nice dashboard. One of those projects that should be much more well known than it is given there are few open source distributed storage solutions. The MFS devs are good but maybe lack the marketing savvy or perhaps just happy where they are.
- mattbillenstein 10y ago
  I also had a lot of issues with Gluster -- when it was broken, it was hard to know what was wrong. With MooseFS, the dashboard would usually give you a good idea, and the logs were pretty good at showing how it was replicating data after a node or disk failed.