5 ms·

found this interesting: https://indico.cern.ch/event/346931/session/4/contribution/62/attachments/684674/940478/beegfs-at-desy.pdf https://indico.cern.ch/event

by poelzi 11y ago

found this interesting: https://indico.cern.ch/event/346931/session/4/contribution/62/attachments/684674/940478/beegfs-at-desy.pdf https://indico.cern.ch/event/346931/session/4/contribution/6... > Plain BeeGFS is as reliable as your hardware currently i would give MooseFS, orangefs or lizardfs a try. Had experience with ceph and glusterfs and both I can not recommend.

pjc50 11y ago
Could you elaborate on this please?
- eis 11y ago
  I'd be also interested in a bit of info why Ceph and GlusterFS disappointed you. I tried GlusterFS a few years ago and it took a good while to figure out the right setup but in the end had disappointing small file performance. Ceph was very interesting but wasn't product ready when I looked at it and since then it somehow never got to a stage where people would rave about it so that alone makes me think it's maybe too complicated. A bit like BTRFS which also for years and years is being developed and sounds really promising but never seems to be finally ready without gotchas.
  - mattbillenstein 11y ago
    I also tried ceph and gluster before settling on moosefs a couple years ago -- gluster was slow for filesystem operations on a lot of files and it would get into a state where some files weren't replicated properly with seemingly no problems with the network for physical servers. Ceph at that time was unstable -- once I had a couple boxes kernel panic in testing, I jumped off that boat immediately. Moosefs is pragmatic -- written in pretty tight C, performant, and the web UI for seeing the state of the cluster and how files are replicating is very nice. We have a small cluster with maybe a dozen nodes running for almost 2 years now with no hiccups...
    - scurvy 11y ago
      Kernel panic? Were you using ceph or cephfs? Ceph block storage shouldn't have any loaded kernel modules.
      - mattbillenstein 11y ago
        We were trying out the filesystem -- it was clearly alpha/beta quality at that time, so it was no surprise it didn't work, and perhaps it's improved in the last two years.
        scurvy 11y ago
        cephfs is clearly marked as not for production. It still has more than its fair share of bugs and has single points of failure. From the very top of the docs page on ceph FS: "Important CephFS currently lacks a robust ‘fsck’ check and repair function. Please use caution when storing important data as the disaster recovery tools are still under development. For more information about using CephFS today, see CephFS for early adopters" RedHat is focusing on making the block and object storage bulletproof. They're content to let the opensource side twiddle away on ceph FS.
    - scurvy 11y ago
      Kernel panic? Were you using ceph or cephfs? Ceph block storage shouldn't have any loaded kernel modules.
  - scurvy 11y ago
    Ceph block storage or CephFS?
birc_a 11y ago
MooseFS and BeeGFS don't really seem to be in the same category of filesystem though. BeeGFS comes from the HPC world where it is all about performance, while MooseFS seems more focused on high reliability even in the face of entire machines coming and going. BeeGFS stores files striped over multiple machines and can use infiniband natively which gives us a system where bandwidth to individual files can reach a bit over 1GB/s (best case) and aggregated bandwidth can reach 30GB/s from a very mixed and unoptimized bioinformatics workload, a lot more when just testing the raw bandwidth. Since BeeGFS uses an underlying filesystem on each storage target you can of course run raid, zfs or whatever it takes to make you comfortable that the individual storage targets aren't going to be lost - which is what it takes for data to be unavailable. If you want some extra reliability in BeeGFS it also supports mirroring so you only lose data if you fully lose two storage targets. We can't really afford to run with full mirroring for our 3PB though. We are very happy with it for our HPC environment but I'm not sure how well it works in an AWS setup.

by poelzi 11y ago

pjc50 11y ago

Could you elaborate on this please?

eis 11y ago
I'd be also interested in a bit of info why Ceph and GlusterFS disappointed you. I tried GlusterFS a few years ago and it took a good while to figure out the right setup but in the end had disappointing small file performance. Ceph was very interesting but wasn't product ready when I looked at it and since then it somehow never got to a stage where people would rave about it so that alone makes me think it's maybe too complicated. A bit like BTRFS which also for years and years is being developed and sounds really promising but never seems to be finally ready without gotchas.
- mattbillenstein 11y ago
  I also tried ceph and gluster before settling on moosefs a couple years ago -- gluster was slow for filesystem operations on a lot of files and it would get into a state where some files weren't replicated properly with seemingly no problems with the network for physical servers. Ceph at that time was unstable -- once I had a couple boxes kernel panic in testing, I jumped off that boat immediately. Moosefs is pragmatic -- written in pretty tight C, performant, and the web UI for seeing the state of the cluster and how files are replicating is very nice. We have a small cluster with maybe a dozen nodes running for almost 2 years now with no hiccups...
  - scurvy 11y ago
    Kernel panic? Were you using ceph or cephfs? Ceph block storage shouldn't have any loaded kernel modules.
    - mattbillenstein 11y ago
      We were trying out the filesystem -- it was clearly alpha/beta quality at that time, so it was no surprise it didn't work, and perhaps it's improved in the last two years.
      - scurvy 11y ago
        cephfs is clearly marked as not for production. It still has more than its fair share of bugs and has single points of failure. From the very top of the docs page on ceph FS: "Important CephFS currently lacks a robust ‘fsck’ check and repair function. Please use caution when storing important data as the disaster recovery tools are still under development. For more information about using CephFS today, see CephFS for early adopters" RedHat is focusing on making the block and object storage bulletproof. They're content to let the opensource side twiddle away on ceph FS.
  - scurvy 11y ago
    Kernel panic? Were you using ceph or cephfs? Ceph block storage shouldn't have any loaded kernel modules.
- scurvy 11y ago
  Ceph block storage or CephFS?

birc_a 11y ago

MooseFS and BeeGFS don't really seem to be in the same category of filesystem though. BeeGFS comes from the HPC world where it is all about performance, while MooseFS seems more focused on high reliability even in the face of entire machines coming and going. BeeGFS stores files striped over multiple machines and can use infiniband natively which gives us a system where bandwidth to individual files can reach a bit over 1GB/s (best case) and aggregated bandwidth can reach 30GB/s from a very mixed and unoptimized bioinformatics workload, a lot more when just testing the raw bandwidth. Since BeeGFS uses an underlying filesystem on each storage target you can of course run raid, zfs or whatever it takes to make you comfortable that the individual storage targets aren't going to be lost - which is what it takes for data to be unavailable. If you want some extra reliability in BeeGFS it also supports mirroring so you only lose data if you fully lose two storage targets. We can't really afford to run with full mirroring for our 3PB though. We are very happy with it for our HPC environment but I'm not sure how well it works in an AWS setup.