3 ms·

> Multi Availability Zone (Multi-AZ), where two database instances are synchronously operated in two different Availability Zones. > The second group of Multi-

> Multi Availability Zone (Multi-AZ), where two database instances are synchronously operated in two different Availability Zones. > The second group of Multi-AZ instances did not failover automatically because the master database instances were disconnected from their standby for a brief time interval immediately before these master database instances’ volumes became stuck. Normally these events are simultaneous. Between the period of time the masters were disconnected from their standbys and the point where volumes became stuck, the masters continued to process transactions without being able to replicate to their standbys. Can someone explain this? I thought the entire point of synchronous replication was that the master doesn't acknowledge that a transaction is committed until the data reaches the slave. That's how it's described in the RDS FAQ: http://aws.amazon.com/rds/faqs/#36 http://aws.amazon.com/rds/faqs/#36

mcpherrinm 14y ago
This just sounds like a race condition gone weird. I assume from this description that the update protocol looks something like this: process-request: Update-local push to sync buddy wait for sync buddy success reply-with-status mark state as synced If the function does the local update and then gets stuck in the state waiting for the buddy to reply, one could imagine the failover daemon not handling that case very well. So while the master might not have acknowledged the transaction, the pair might get jammed trying to complete it. EBS is a particularily complicated piece of software, and RDS is another layer of complication built on top of that. Bugs clearly happen, and it's an unfortunate state of affairs.
redditmigrant 14y ago
I think aws uses drbd (http://www.drbd.org/ http://www.drbd.org/) for replicating the writes to the standby. Among the replication modes of drbd suggested here: http://www.drbd.org/users-guide-emb/s-replication-protocols.html http://www.drbd.org/users-guide-emb/s-replication-protocols.... I suspect they probably use B and landed in a state where packets got delivered but werent written on the standby.

mcpherrinm 14y ago

This just sounds like a race condition gone weird. I assume from this description that the update protocol looks something like this: process-request: Update-local push to sync buddy wait for sync buddy success reply-with-status mark state as synced If the function does the local update and then gets stuck in the state waiting for the buddy to reply, one could imagine the failover daemon not handling that case very well. So while the master might not have acknowledged the transaction, the pair might get jammed trying to complete it. EBS is a particularily complicated piece of software, and RDS is another layer of complication built on top of that. Bugs clearly happen, and it's an unfortunate state of affairs.

redditmigrant 14y ago

I think aws uses drbd (http://www.drbd.org/ http://www.drbd.org/) for replicating the writes to the standby. Among the replication modes of drbd suggested here: http://www.drbd.org/users-guide-emb/s-replication-protocols.html http://www.drbd.org/users-guide-emb/s-replication-protocols.... I suspect they probably use B and landed in a state where packets got delivered but werent written on the standby.