5 ms·

I'm not a GlusterFS expert, and haven't used it before, but you should know that most consensus algorithms (Paxos, Raft, etc) only function reliably with an odd

I'm not a GlusterFS expert, and haven't used it before, but you should know that most consensus algorithms (Paxos, Raft, etc) only function reliably with an odd number of nodes. I have to wonder if your problems were mostly self-inflicted from having 2 nodes. Of course, any network partition in a 2-node cluster has a huge potential for data corruption, as each node now thinks it is the master (split-brain). In a 3-node cluster, any system with a decent consensus algorithm (to be clear, I'm not sure if GlusterFS has one) would know that during a partition the cluster can only continue to operate if at least 2 nodes can communicate with each other to elect a new master.

tveita 11y ago
> Of course, any network partition in a 2-node cluster has a huge potential for data corruption, as each node now thinks it is the master (split-brain). This is not a given, the cluster can and should refuse to operate if it can not get a majority vote for the master, if that is required to prevent data corruption. With two nodes that means both nodes must be active and reachable, with three nodes one node can fail.
gamegod 11y ago
I didn't do the initial configuration myself but I wouldn't rule out this kind of gross, self-inflicted problem. One has to wonder why the software even lets you run it with an unsafe number of nodes though... (Lots of distributed software is guilty of this...)
- toredash 11y ago
  Uh, well if you base your GlusterFS experience on that, no wonder you have a bad view on the technology. I can only assume that your setup only used two nodes, which is not really supported, and of course will cause split-brain problems. And two bricks, really? GlusterFS is best aimed for scale both on the server and client side, two bricks is not really scale.
  - gamegod 11y ago
    There's probably a good case study to be made here on why so many people have had bad experiences with GlusterFS. If the problem is that it's too easy to set it up in a fragile way, then they need to fix that, because otherwise you're going to end up with a lot unhappy users (as you can see in the other comments). Still, if I can't make it work well on a reasonably small scale, why would I expect it to work any better when I scale it up?
    - toredash 11y ago
      I would suggest reading the documentation and understanding the architecture. It works well for many applications but some are less ideal. Maybe yours was a corner case? I don't know. But it's used by many, for instance Facebook (https://www.socallinuxexpo.org/scale/14x/presentations/scaling-glusterfs-facebook https://www.socallinuxexpo.org/scale/14x/presentations/scali...).
oh_sigh 11y ago
But then in a split with a 3-node cluster, there is now a 2-node cluster in charge, and what happens if another partition happens?
- illumin8 11y ago
  Typically, clusters without a majority (2 out of 3, 3 out of 5, 4 out of 7, etc) of nodes present will shut themselves down to prevent data corruption.
- duaneb 11y ago
  It depends on the underlying algorithm. These risks are bounded, not solved.

tveita 11y ago

> Of course, any network partition in a 2-node cluster has a huge potential for data corruption, as each node now thinks it is the master (split-brain). This is not a given, the cluster can and should refuse to operate if it can not get a majority vote for the master, if that is required to prevent data corruption. With two nodes that means both nodes must be active and reachable, with three nodes one node can fail.

gamegod 11y ago

I didn't do the initial configuration myself but I wouldn't rule out this kind of gross, self-inflicted problem. One has to wonder why the software even lets you run it with an unsafe number of nodes though... (Lots of distributed software is guilty of this...)

toredash 11y ago
Uh, well if you base your GlusterFS experience on that, no wonder you have a bad view on the technology. I can only assume that your setup only used two nodes, which is not really supported, and of course will cause split-brain problems. And two bricks, really? GlusterFS is best aimed for scale both on the server and client side, two bricks is not really scale.
- gamegod 11y ago
  There's probably a good case study to be made here on why so many people have had bad experiences with GlusterFS. If the problem is that it's too easy to set it up in a fragile way, then they need to fix that, because otherwise you're going to end up with a lot unhappy users (as you can see in the other comments). Still, if I can't make it work well on a reasonably small scale, why would I expect it to work any better when I scale it up?
  - toredash 11y ago
    I would suggest reading the documentation and understanding the architecture. It works well for many applications but some are less ideal. Maybe yours was a corner case? I don't know. But it's used by many, for instance Facebook (https://www.socallinuxexpo.org/scale/14x/presentations/scaling-glusterfs-facebook https://www.socallinuxexpo.org/scale/14x/presentations/scali...).

oh_sigh 11y ago

But then in a split with a 3-node cluster, there is now a 2-node cluster in charge, and what happens if another partition happens?

illumin8 11y ago
Typically, clusters without a majority (2 out of 3, 3 out of 5, 4 out of 7, etc) of nodes present will shut themselves down to prevent data corruption.
duaneb 11y ago
It depends on the underlying algorithm. These risks are bounded, not solved.