4 ms·

> HA Postgres is an extremely painful proposition. Does anyone here know how Amazon RDS's HA setup, particularly their multi-AZ option, works? That seems to be

> HA Postgres is an extremely painful proposition. Does anyone here know how Amazon RDS's HA setup, particularly their multi-AZ option, works? That seems to be a switch that the AWS customer can just turn on. Do they have a proprietary implementation, even for non-Aurora Postgres?

threeseed 6y ago
They basically have built a proprietary, distributed block store. And on top of this they have layered PostgreSQL, MySQL, MongoDB, Cassandra etc. I doubt they will never release the code for it since it's very much a competitive advantage.
- devit 6y ago
  So if the hardware running the database is suddenly destroyed they try to start another instance really fast? That seems inferior to having multiple sync replicas ready to take over without having to start a process and replay the WAL. Also, such an HA block store seems very easy to replicate ( I'd guess there would be something open source already), not much of a competitive advantage.
  - redis_mlc 6y ago
    - using block-level replication allows them to support multiple databases in a common way - block-level replication can be more reliable in the long run operationally than some types of database replication, especially MySQL back in the day - block-level replication has more scalable support staff available than hiring DBAs to fix database replication problems - programming for all the edge cases is something that is a competitive advantage - no licensing required for it - you can probably guess which Open Source project it's based on Source: DBA, worked there.
    - hashhar 6y ago
      Just gonna put this here for people who might be wondering. Not too many block-level replication implementations exist. https://www.linbit.com/drbd/ https://www.linbit.com/drbd/ NB: It is Free software, don't be alarmed by the domain name.
    - bratao 6y ago
      Linstor can do that. I use in Kubernetes and it is incredible!
  - aeyes 6y ago
    There are two types of failover in RDS: - Single instance, if the instance dies they start a new one and mount the same storage. This can take some time, in general under 5min but I have seen it take 45min, especially for the large instance types. - Multi A-Z, they run a hot standby with replicated and physically separated storage. Failover takes about a minute. The replication happens at storage level, every write has to be acknowledged by both availability zones. I'm not sure if Postgres is always running or if it gets started when failing over. I guess you could replicate this using drbd.
- crgwbr 6y ago
  That’s how the Aurora DBs work, but I believe standard RDS Postgres uses a more typical method of WAL replication.
  - jpgvm 6y ago
    For read replicas yes but not for the standby master. They give you access to pg_stat_replication so this is quite easy to verify.
aeyes 6y ago
Here is pretty much the most detailed post about how it works you'll be able to find in public: https://aws.amazon.com/blogs/database/amazon-rds-under-the-hood-multi-az/ https://aws.amazon.com/blogs/database/amazon-rds-under-the-h... They basically do replication at the storage layer. Each write has to be acknowledged by both the primary and secondary EBS volume.

threeseed 6y ago

They basically have built a proprietary, distributed block store. And on top of this they have layered PostgreSQL, MySQL, MongoDB, Cassandra etc. I doubt they will never release the code for it since it's very much a competitive advantage.

devit 6y ago
So if the hardware running the database is suddenly destroyed they try to start another instance really fast? That seems inferior to having multiple sync replicas ready to take over without having to start a process and replay the WAL. Also, such an HA block store seems very easy to replicate ( I'd guess there would be something open source already), not much of a competitive advantage.
- redis_mlc 6y ago
  - using block-level replication allows them to support multiple databases in a common way - block-level replication can be more reliable in the long run operationally than some types of database replication, especially MySQL back in the day - block-level replication has more scalable support staff available than hiring DBAs to fix database replication problems - programming for all the edge cases is something that is a competitive advantage - no licensing required for it - you can probably guess which Open Source project it's based on Source: DBA, worked there.
  - hashhar 6y ago
    Just gonna put this here for people who might be wondering. Not too many block-level replication implementations exist. https://www.linbit.com/drbd/ https://www.linbit.com/drbd/ NB: It is Free software, don't be alarmed by the domain name.
  - bratao 6y ago
    Linstor can do that. I use in Kubernetes and it is incredible!
- aeyes 6y ago
  There are two types of failover in RDS: - Single instance, if the instance dies they start a new one and mount the same storage. This can take some time, in general under 5min but I have seen it take 45min, especially for the large instance types. - Multi A-Z, they run a hot standby with replicated and physically separated storage. Failover takes about a minute. The replication happens at storage level, every write has to be acknowledged by both availability zones. I'm not sure if Postgres is always running or if it gets started when failing over. I guess you could replicate this using drbd.
crgwbr 6y ago
That’s how the Aurora DBs work, but I believe standard RDS Postgres uses a more typical method of WAL replication.
- jpgvm 6y ago
  For read replicas yes but not for the standby master. They give you access to pg_stat_replication so this is quite easy to verify.

aeyes 6y ago

Here is pretty much the most detailed post about how it works you'll be able to find in public: https://aws.amazon.com/blogs/database/amazon-rds-under-the-hood-multi-az/ https://aws.amazon.com/blogs/database/amazon-rds-under-the-h... They basically do replication at the storage layer. Each write has to be acknowledged by both the primary and secondary EBS volume.