4 ms·
Sadly, even simple concepts like STONITH are hard to get right. I believe it was GitHub that had an outage because both db nodes shot each other, but their netw
by lucian1900 13y ago
Sadly, even simple concepts like STONITH are hard to get right. I believe it was GitHub that had an outage because both db nodes shot each other, but their network was extremely slow because of some fault (which caused the initial problem as well) and both nodes received the STONITH message from the other at similar times, long after they each timed out waiting for a response.
Distributed systems are hard.
- derekp7 13y agoOne thing to keep in mind, is clustering will never make your app 100% available. The best it can do is add 1 "9" to your uptime (so you are already at 99.9 percent, it can get you to 99.99 percent uptime, etc). But one thing that it should NEVER do is corrupt your data. If you have a service outage, yet your data was kept safe, then the cluster still did it's job.
- MichaelGG 13y agoIs this for scenarios where there's no shared storage medium, so IO fencing isn't a solution? I can see for a DRBD solution how you need some real way of ensuring only one node is up. It just feels like STONITH is a really ugly hack and would be better solved via other quorum solutions, even if it means adding a witness system.
- derekp7 13y agoNormally when you have shared storage, you can use that shared storage as a quorum device (i.e., via an exclusive scsi lock). If you have something like DRBD, then you still have the issue where each node can't see each other, but an outside application writing to the database served up by the cluster can see both nodes -- and if each node wants to bring up it's shared IP address, some writes will go to one node, some to the other. Then you have the database on each node not having all the current data (even if it isn't technically "corrupted").