6 ms·

Is anyone here using GlusterFS in production? I'd be interested to hear about how people have recovered from critical hardware failure when using it...

Nux 13y ago
Using it in production (v3.4.0), but haven't had a critical fail yet. The fact that I keep my "bricks" on top of RAID(6) also helped with that.
- deleted 13y ago
  [deleted]
jacquesm 13y ago
I used a very early version of GlusterFS on a 5 node cluster and never lost any data (about 32T worth, quite a bit at the time). What I liked most about it was that it produced storage nodes that could be inspected with the regular toolset, even if something would go down (which it never did) it would definitely have been possible to recover by looking at the crashed storage. Having the 'bricks' use the regular filesystems was a very smart decision, it means you don't end up in binary hell when things go south. Split brain was one of the weaker points back in the days, I am not current with the software so no idea if and how much this has improved, someone with more recent experience would be a better guide there.
- notacoward 13y ago
  Sadly, split brain is still a bit of a sore spot. It probably will be as long as GlusterFS continues to use the "fan out" replication approach, though past and future enhancements around quorum should help too. On the other hand, it's great to get such positive feedback about the "everything is regular files" doctrine. It has certainly been a bit painful sometimes, especially in terms of performance, but the feeling has always been that the ops advantages are worth it. Nice to see someone else agrees. (Disclaimer: I'm a GlusterFS developer)
  - jacquesm 13y ago
    Hah, cool. Is 'dtor' still with the team? If so please say hi to him.
jamescun 13y ago
I used GlusterFS a couple of years ago (2011) for a hosting project, we had about 300 or so medium-large PHP sites based on top of a GlusterFS cluster. It ran beautifully but the pain points where it fell down was small file performance and its configuration system (I think this has been made much nicer in more recent versions).
- growse 13y ago
  I had a similar experience trying to build a highly-available wordpress infrastructure a few years back. The terrible latency of gluster completely killed any chance of getting decent web performance out of it. Ended up implementing OCFS2 on DRBD, much better.
  - velodrome 13y ago
    Were you serving the PHP files off of Gluster? Or just the static files?
    - stevekemp 13y ago
      I had a client running PHP from a gluster storage, split across two nodes. Performance was horrific, although some of the sites were improved via Varnish we eventually had to move to local copies of files & rsync. Not pleasant. Gluster is nice, but you have to accept the fact that it is slow, and upgrades are painful. (We had issues where clients and servers had to be updated at the same time, to avoid interoperability issues.)
amatix 13y ago
We run it in some 2-node configurations with 4-6 bricks/node (Gluster is configured as distribute+replicate). Replacing disks isn't a big deal, there's a replace-brick command which migrates the data away from a brick to a new one, or for hard fails you can just drop in a new disk, do replace-brick and force a heal to re-replicate data. A node going down and coming back is straightforward, the self-heal-daemon and disk activity get the "missed" changes re-replicated (or force it via heal) -- everything keeps working in the meantime. Native clients ignore "down" nodes without needing intervention (for NFS clients use a shared IP and fail that over to any "up" gluster node). Performance gets better with every release, but the speed of `ls` (Gluster uses `stat()` to trigger replication/checking) is still woeful at times. 3.5 will hopefully make that a lot better. In general, Gluster is not the right answer if you have lots of small files.
snorkel 13y ago
I've used it on production in EC2. The setup essentially replicated the underlying files to at least two nodes so if a host failed then the volume was still entirely available automatically, and when the dead host recovered it would replicate on read. It was great for mostly read workloads, but sluggish when the workload involves a lot of writes due to replication costs.
syrneus 13y ago
We run hundreds of GlusterFS clusters in production on EC2. We're currently on 3.0 and in the process of fully migrating to 3.4 (and maybe 3.5 one day). Our primary use case for Gluster is in serving persistent filesystems for Drupal. Our customers store potentially millions of files on their GlusterFS clusters. We've built a number of tools/processes to help protect Gluster against failures in EC2 (for instance fencing network traffic at the iptables layer to help protect GlusterFS clients from hanging talking to down nodes), as well as to help our team perform common tasks (resizing clusters, moving customers from cluster to cluster, etc.). We haven't necessarily hit blocker issues recovering from underlying hardware fails, but our team is definitely very experienced with many possible failure modes. Overall GlusterFS has been very reliable over the years and our research has shown it is the best option out there for when our customers can't use something such as S3 directly. If you want more details or would love to hack on a 8000+ node EC2 cluster running things such as GlusterFS feel free to ping me.
- deleted 13y ago
  [deleted]
- orf 13y ago
  Just out of interest why don't you use S3 for this? Amazon provides a few options for scalable storage, is it cheaper to roll your own ontop of EC2?
  - pfg 13y ago
    Basically, they need EBS without the single-instance (writable) mount limit, if they're using it to sync Drupal installations (php files, temp folders, etc.) across multiple web servers. S3 isn't really intended for that kind of workload (it would be possible to use something like s3fs, but I don't think that will give you a POSIX-compatible file system which is required for Drupal, and it's probably too slow as well (lots of small files, etc.))
  - syrneus 13y ago
    Many of our customers do use S3 and we make use of S3 extensively ourselves. However, Drupal often expects to operate on a POSIX compatible filesystem. Drupal 7 does support PHP file streams which can be configured to use S3, but not every Drupal module follows the best practices. Plus, we support every flavor of Drupal under the sun (including custom code). All of our enterprise customers receive a highly available setup running on multiple nodes--thus, we have the need for a persistent filesystem attached to multiple EC2 instances. We utilize GlusterFS to ensure all of our clients have the filesystem capabilities their apps may need.
raggiskula 13y ago
Well, we used it for storage backend for VMs few years back. For some reasons it usually failed on high load, resulting in bad split-brains, extremely slow IO and loss of data. Very bad. We contacted Red Hat regarding that and got comments that gluster was not VM ready yet. We ditched gluster and went for plane proven old iSCSI.
- Rantenki 13y ago
  This. We had the same failure mode, on a cluster with dozens of nodes and thousands of VMs. http://thr3ads.net/gluster-users/2011/06/480298-Enomaly-users-please-read http://thr3ads.net/gluster-users/2011/06/480298-Enomaly-user... This bug (and others) would happen when moving files and/or folders over top of existing ones, ie: mkdir -p foo/bar/baz && \ mkdir -p foo/bar/tmp && \ touch foo/bar/baz/file foo/bar/tmp/file && \ mv foo/bar/tmp/file foo/bar/baz/ && \ rm -rf foo/bar/tmp This was a method to replace an existing lockfile. Mercurial uses some similar code. The expected behavior would be that foo/bar/tmp/file replaced /foo/bar/baz/file. Instead, the outcome was that there was a race in Gluster where it got confused about which version of 'file' was correct, and it would end up with a split brain between the two nodes. This would be exacerbated by a node failure, but didn't always require one. Heavy load seemed to make the failure more likely. We couldn't replicate the same bug moving files in the same folder, it was the subfolders in the same gluster fs that seemed to cause the issue. The frustrating part was how gluster pointed fingers at the file operation being incompatible, despite advertising a posix compatible filesystem. Apparently the bug is fixed, but we moved off gluster, never to return. Also, filesystems are hard, I get it, so no hard feelings :)
jamesjporter 13y ago
We used to use gluster in production but had similar problems to raggiskula and are currently in the process of moving away from it.

Nux 13y ago

Using it in production (v3.4.0), but haven't had a critical fail yet. The fact that I keep my "bricks" on top of RAID(6) also helped with that.

deleted 13y ago
[deleted]

jacquesm 13y ago

I used a very early version of GlusterFS on a 5 node cluster and never lost any data (about 32T worth, quite a bit at the time). What I liked most about it was that it produced storage nodes that could be inspected with the regular toolset, even if something would go down (which it never did) it would definitely have been possible to recover by looking at the crashed storage. Having the 'bricks' use the regular filesystems was a very smart decision, it means you don't end up in binary hell when things go south. Split brain was one of the weaker points back in the days, I am not current with the software so no idea if and how much this has improved, someone with more recent experience would be a better guide there.

notacoward 13y ago
Sadly, split brain is still a bit of a sore spot. It probably will be as long as GlusterFS continues to use the "fan out" replication approach, though past and future enhancements around quorum should help too. On the other hand, it's great to get such positive feedback about the "everything is regular files" doctrine. It has certainly been a bit painful sometimes, especially in terms of performance, but the feeling has always been that the ops advantages are worth it. Nice to see someone else agrees. (Disclaimer: I'm a GlusterFS developer)
- jacquesm 13y ago
  Hah, cool. Is 'dtor' still with the team? If so please say hi to him.

jamescun 13y ago

I used GlusterFS a couple of years ago (2011) for a hosting project, we had about 300 or so medium-large PHP sites based on top of a GlusterFS cluster. It ran beautifully but the pain points where it fell down was small file performance and its configuration system (I think this has been made much nicer in more recent versions).

growse 13y ago
I had a similar experience trying to build a highly-available wordpress infrastructure a few years back. The terrible latency of gluster completely killed any chance of getting decent web performance out of it. Ended up implementing OCFS2 on DRBD, much better.
- velodrome 13y ago
  Were you serving the PHP files off of Gluster? Or just the static files?
  - stevekemp 13y ago
    I had a client running PHP from a gluster storage, split across two nodes. Performance was horrific, although some of the sites were improved via Varnish we eventually had to move to local copies of files & rsync. Not pleasant. Gluster is nice, but you have to accept the fact that it is slow, and upgrades are painful. (We had issues where clients and servers had to be updated at the same time, to avoid interoperability issues.)

amatix 13y ago

We run it in some 2-node configurations with 4-6 bricks/node (Gluster is configured as distribute+replicate). Replacing disks isn't a big deal, there's a replace-brick command which migrates the data away from a brick to a new one, or for hard fails you can just drop in a new disk, do replace-brick and force a heal to re-replicate data. A node going down and coming back is straightforward, the self-heal-daemon and disk activity get the "missed" changes re-replicated (or force it via heal) -- everything keeps working in the meantime. Native clients ignore "down" nodes without needing intervention (for NFS clients use a shared IP and fail that over to any "up" gluster node). Performance gets better with every release, but the speed of `ls` (Gluster uses `stat()` to trigger replication/checking) is still woeful at times. 3.5 will hopefully make that a lot better. In general, Gluster is not the right answer if you have lots of small files.

snorkel 13y ago

I've used it on production in EC2. The setup essentially replicated the underlying files to at least two nodes so if a host failed then the volume was still entirely available automatically, and when the dead host recovered it would replicate on read. It was great for mostly read workloads, but sluggish when the workload involves a lot of writes due to replication costs.

syrneus 13y ago

We run hundreds of GlusterFS clusters in production on EC2. We're currently on 3.0 and in the process of fully migrating to 3.4 (and maybe 3.5 one day). Our primary use case for Gluster is in serving persistent filesystems for Drupal. Our customers store potentially millions of files on their GlusterFS clusters. We've built a number of tools/processes to help protect Gluster against failures in EC2 (for instance fencing network traffic at the iptables layer to help protect GlusterFS clients from hanging talking to down nodes), as well as to help our team perform common tasks (resizing clusters, moving customers from cluster to cluster, etc.). We haven't necessarily hit blocker issues recovering from underlying hardware fails, but our team is definitely very experienced with many possible failure modes. Overall GlusterFS has been very reliable over the years and our research has shown it is the best option out there for when our customers can't use something such as S3 directly. If you want more details or would love to hack on a 8000+ node EC2 cluster running things such as GlusterFS feel free to ping me.

deleted 13y ago
[deleted]
orf 13y ago
Just out of interest why don't you use S3 for this? Amazon provides a few options for scalable storage, is it cheaper to roll your own ontop of EC2?
- pfg 13y ago
  Basically, they need EBS without the single-instance (writable) mount limit, if they're using it to sync Drupal installations (php files, temp folders, etc.) across multiple web servers. S3 isn't really intended for that kind of workload (it would be possible to use something like s3fs, but I don't think that will give you a POSIX-compatible file system which is required for Drupal, and it's probably too slow as well (lots of small files, etc.))
- syrneus 13y ago
  Many of our customers do use S3 and we make use of S3 extensively ourselves. However, Drupal often expects to operate on a POSIX compatible filesystem. Drupal 7 does support PHP file streams which can be configured to use S3, but not every Drupal module follows the best practices. Plus, we support every flavor of Drupal under the sun (including custom code). All of our enterprise customers receive a highly available setup running on multiple nodes--thus, we have the need for a persistent filesystem attached to multiple EC2 instances. We utilize GlusterFS to ensure all of our clients have the filesystem capabilities their apps may need.

raggiskula 13y ago

Well, we used it for storage backend for VMs few years back. For some reasons it usually failed on high load, resulting in bad split-brains, extremely slow IO and loss of data. Very bad. We contacted Red Hat regarding that and got comments that gluster was not VM ready yet. We ditched gluster and went for plane proven old iSCSI.

Rantenki 13y ago
This. We had the same failure mode, on a cluster with dozens of nodes and thousands of VMs. http://thr3ads.net/gluster-users/2011/06/480298-Enomaly-users-please-read http://thr3ads.net/gluster-users/2011/06/480298-Enomaly-user... This bug (and others) would happen when moving files and/or folders over top of existing ones, ie: mkdir -p foo/bar/baz && \ mkdir -p foo/bar/tmp && \ touch foo/bar/baz/file foo/bar/tmp/file && \ mv foo/bar/tmp/file foo/bar/baz/ && \ rm -rf foo/bar/tmp This was a method to replace an existing lockfile. Mercurial uses some similar code. The expected behavior would be that foo/bar/tmp/file replaced /foo/bar/baz/file. Instead, the outcome was that there was a race in Gluster where it got confused about which version of 'file' was correct, and it would end up with a split brain between the two nodes. This would be exacerbated by a node failure, but didn't always require one. Heavy load seemed to make the failure more likely. We couldn't replicate the same bug moving files in the same folder, it was the subfolders in the same gluster fs that seemed to cause the issue. The frustrating part was how gluster pointed fingers at the file operation being incompatible, despite advertising a posix compatible filesystem. Apparently the bug is fixed, but we moved off gluster, never to return. Also, filesystems are hard, I get it, so no hard feelings :)

jamesjporter 13y ago

We used to use gluster in production but had similar problems to raggiskula and are currently in the process of moving away from it.