4 ms·

What would be a more fool-proof distributed storage? I tried GlusterFS briefly, but also heard some meltdown stories about it. It almost seems like if you need

by amq 9y ago

What would be a more fool-proof distributed storage? I tried GlusterFS briefly, but also heard some meltdown stories about it. It almost seems like if you need some persistent storage running plain NFS would be the safest thing if you don't have a dedicated team.

wmf 9y ago
That's what I've done. Hardware RAID + XFS + NFS + backups, done. It wasn't HA, but at least I understood how to get it back up.
- majidazimi 9y ago
  I can't emphasize this more. Being able to comprehend the system is my first design choice, no matter how the alternatives are great/stable/fast, ... You are going to pay the price some day as long as you don't know how stuff works or interacts with each other.
- snark42 9y ago
  > That's what I've done. Hardware RAID + XFS + NFS + backups, done. Even if this was HA, it doesn't scale to HPC type needs. You end up with a single node bottleneck for all data you need. You also would have some sort of direct attached storage all connected to a single node. Lustre/Gluster/Ceph/GFS and maybe HDFS (ideally with commercial MapR type NFS access) are probably the only viable options in the HPC use case. If you have a lot of money to burn Isilon/NFS is an option as well.
  - wmf 9y ago
    Yep, I haven't worked in a HPC/big data environment but neither are the people who wrote the original article. They have three OSDs and only two(?) servers accessing them.
- clon 9y ago
  > That's what I've done. Hardware RAID + XFS + NFS + backups, done. It wasn't HA, but at least I understood how to get it back up. Looks like in this case you never actually needed what Ceph brings to the table - for a tremendous cost in complexity (and fragility, from personal anecdotal evidence) you can allow your dataset and -load to grow beyond the physical capabilities of a single node. The HA argument in less demanding settings ends up being a cargo cult mostly. Your setup will probably produce higher reliability figures than running Ceph, assuming the lack of some serious engineering capabilities.
deleted 9y ago
[deleted]
tene 9y ago
The more-fool-proof option for any distributed storage system is to not deploy multiple package upgrades across your entire fleet without any testing. This entire problem would have been avoided (for any distributed storage system) if they had first tested the upgrade on a single server before rolling it across their entire fleet. If you've got different code sitting around on disk ready for a surprise upgrade whenever something eventually someday gets restarted, you're going to be in for a bad time. > It turned out that all three systems had been updated a few times without restarting the OSD. No OSD could start anymore. We kept the last two OSD running (this turned out to be a mistake). The file servers, running Gentoo, also had a profile update done by another administrator. If you don't have the resources to perform safe upgrades, the fool-proof option is to pay a vendor to run your storage for you. I agree that if you want to run a reliable distributed storage system, you need a professional sysadmin who has enough time to maintain it safely. I further agree that if you don't have any professionals who are funded to dedicate at least some time to this, you'll probably have a lot less failure by just running a big dedicated node providing NFS, iSCSI, SMB, or whatever. I can't think of any software that I'd call fool-proof under "We've upgraded multiple versions without any testing, we have no systems in a known-good state, and we don't have any way to actually revert back to a known-good state".
- mjevans 9y ago
  A /lot/ of times the issue is one of budget. It's //really// nice to actually have a budget so you can setup a testing environment and actually validate the things you're about to do to your production environment.
fulafel 9y ago
Also, take a step back and ask if you really need distributed storage of the OS-mounted filesystem variety. Userspace file storage can be fundamentally less fragile and easier to understand. You really want it to be easy to understand and troubleshoot if you aim for high availability.
riffic 9y ago
Local storage would be advantageous here, and for most cases. VMs are cattle, not pets, and should be redeployable in the event the HV is having issues.

by amq 9y ago

wmf 9y ago

That's what I've done. Hardware RAID + XFS + NFS + backups, done. It wasn't HA, but at least I understood how to get it back up.

majidazimi 9y ago
I can't emphasize this more. Being able to comprehend the system is my first design choice, no matter how the alternatives are great/stable/fast, ... You are going to pay the price some day as long as you don't know how stuff works or interacts with each other.
snark42 9y ago
> That's what I've done. Hardware RAID + XFS + NFS + backups, done. Even if this was HA, it doesn't scale to HPC type needs. You end up with a single node bottleneck for all data you need. You also would have some sort of direct attached storage all connected to a single node. Lustre/Gluster/Ceph/GFS and maybe HDFS (ideally with commercial MapR type NFS access) are probably the only viable options in the HPC use case. If you have a lot of money to burn Isilon/NFS is an option as well.
- wmf 9y ago
  Yep, I haven't worked in a HPC/big data environment but neither are the people who wrote the original article. They have three OSDs and only two(?) servers accessing them.
clon 9y ago
> That's what I've done. Hardware RAID + XFS + NFS + backups, done. It wasn't HA, but at least I understood how to get it back up. Looks like in this case you never actually needed what Ceph brings to the table - for a tremendous cost in complexity (and fragility, from personal anecdotal evidence) you can allow your dataset and -load to grow beyond the physical capabilities of a single node. The HA argument in less demanding settings ends up being a cargo cult mostly. Your setup will probably produce higher reliability figures than running Ceph, assuming the lack of some serious engineering capabilities.

deleted 9y ago

[deleted]

tene 9y ago

The more-fool-proof option for any distributed storage system is to not deploy multiple package upgrades across your entire fleet without any testing. This entire problem would have been avoided (for any distributed storage system) if they had first tested the upgrade on a single server before rolling it across their entire fleet. If you've got different code sitting around on disk ready for a surprise upgrade whenever something eventually someday gets restarted, you're going to be in for a bad time. > It turned out that all three systems had been updated a few times without restarting the OSD. No OSD could start anymore. We kept the last two OSD running (this turned out to be a mistake). The file servers, running Gentoo, also had a profile update done by another administrator. If you don't have the resources to perform safe upgrades, the fool-proof option is to pay a vendor to run your storage for you. I agree that if you want to run a reliable distributed storage system, you need a professional sysadmin who has enough time to maintain it safely. I further agree that if you don't have any professionals who are funded to dedicate at least some time to this, you'll probably have a lot less failure by just running a big dedicated node providing NFS, iSCSI, SMB, or whatever. I can't think of any software that I'd call fool-proof under "We've upgraded multiple versions without any testing, we have no systems in a known-good state, and we don't have any way to actually revert back to a known-good state".

mjevans 9y ago
A /lot/ of times the issue is one of budget. It's //really// nice to actually have a budget so you can setup a testing environment and actually validate the things you're about to do to your production environment.

fulafel 9y ago

Also, take a step back and ask if you really need distributed storage of the OS-mounted filesystem variety. Userspace file storage can be fundamentally less fragile and easier to understand. You really want it to be easy to understand and troubleshoot if you aim for high availability.

riffic 9y ago

Local storage would be advantageous here, and for most cases. VMs are cattle, not pets, and should be redeployable in the event the HV is having issues.