5 ms·

> The exact second DB2 detects an issue with US-East, it switches over to the standby server How does it do this? This is actually a very difficult problem be

> The exact second DB2 detects an issue with US-East, it switches over to the standby server How does it do this? This is actually a very difficult problem because the monitoring system has to determine whether the primary site is down or whether its own network is experiencing trouble. And once you perform the failover, the old master might not learn that it has lost its master role, and may continue to serve requests to clients. Systems with automated failover usually use a lock service like Google's Chubby. For most folks, it's better to have a manual failover script that the oncall engineer can run after diagnosing the issue. Automated failover requires a lot of extra complexity in your systems. There's the real risk of total service failure when the lock service goes down. And there are lots of interesting failure modes in the failover process. For a startup on a tight budget, it's probably not worth it just to change 30 minutes of downtime into 1 minute.

ww520 16y ago
Exactly right. Network partition is a hard problem in automatic failover of replicated system. You don't want the standby to become master unless it can be sure the primary master is absolutely down. It's difficult to unwind the mess if two masters are active and taking changes. In high availability system design, the secondary node literally has to shut down the primary's power (called the Shoot-At-The-Head technique) to ensure it's really down when it's not responding via network. Of course over long distance cross-datacenter replication, shutting down power remotely is not reliable. In the last HA clusters I built, the failover between datacenters is done via manual decision. It means there could be a 15 minutes to 30 minutes window to do the manual failover, but it's an acceptable risk since datacenter failure is rare, like AWS failure once in a blue moon.
- morganpyne 16y ago
  I remember when I first used HA-Linux in a project being highly amused when I came across the acronym STONITH and discovering that it meant "Shoot The Other Node In The Head" :-) http://www.linux-ha.org/wiki/STONITH http://www.linux-ha.org/wiki/STONITH All jokes aside, it is indeed a very important concept when dealing with high availability.

ww520 16y ago

Exactly right. Network partition is a hard problem in automatic failover of replicated system. You don't want the standby to become master unless it can be sure the primary master is absolutely down. It's difficult to unwind the mess if two masters are active and taking changes. In high availability system design, the secondary node literally has to shut down the primary's power (called the Shoot-At-The-Head technique) to ensure it's really down when it's not responding via network. Of course over long distance cross-datacenter replication, shutting down power remotely is not reliable. In the last HA clusters I built, the failover between datacenters is done via manual decision. It means there could be a 15 minutes to 30 minutes window to do the manual failover, but it's an acceptable risk since datacenter failure is rare, like AWS failure once in a blue moon.

morganpyne 16y ago
I remember when I first used HA-Linux in a project being highly amused when I came across the acronym STONITH and discovering that it meant "Shoot The Other Node In The Head" :-) http://www.linux-ha.org/wiki/STONITH http://www.linux-ha.org/wiki/STONITH All jokes aside, it is indeed a very important concept when dealing with high availability.