4 ms·

Self-hosting MariaDB with multi-master replication. Works pretty well for my cases. Have a friend that was surprised by the AWS pricing changes on Aurora (they

Self-hosting MariaDB with multi-master replication. Works pretty well for my cases. Have a friend that was surprised by the AWS pricing changes on Aurora (they charge per query now instead of bandwidth/usage?)

vira28 5y ago
multi-master with conflict resolution is non-trivial. Do you have automated failover set up? Also, didn't know about Aurora's per query pricing. That is crazy.
- toast0 5y ago
  It's been a while since I ran MySQL in prod, but a multi-master setup where all writes go to one of them (and other is set to read-only, and your grants respect read-only) is super operable. If you have your config layer key off of read-only status and you kill existing client connections when you switch from read-write to read-only, it's a pretty small effort to switch masters. I tend to leave the decision to switch masters to humans, and just automate the process. Unless you have bulletproof conflict resolution/reconciliation, automating failover is inviting a conflict that leads to major pain and a large effort to fix. Better to have a few minutes of write downtime while waiting for someone to make the decision (which includes making sure the dead host is truely dead and not just split brained)
- juangacovas 5y ago
  Sorry, I meant multi-source replication (different masters, different DBs to same slave(s)).
starwatch 5y ago
We ran on this set up for a couple of years. We even migrated from linode to AWS ec2 instances (that was a fun problem!). To start with Galera was okay, but as our traffic picked up things became sketchy. As traffic grew we'd encounter evermore outages - it seemed that some percentage of requests between nodes would be dropped (despite AWS's insistence to the contrary) and the nodes couldn't seem to gracefully recover. With the rapid growth, and tiny team we didn't have enough time to fully get to the bottom of it sadly. The final straw for Galera for us was when we started exploring multi-region replication (us-east-1 and Singapore). The bandwidth between regions was just not high enough to support a cluster. Aurora offered all that and more (though MariaDB to MySQL was a bit fiddly). Since migrating life (and uptime) has improved immeasurably.

vira28 5y ago

multi-master with conflict resolution is non-trivial. Do you have automated failover set up? Also, didn't know about Aurora's per query pricing. That is crazy.

toast0 5y ago
It's been a while since I ran MySQL in prod, but a multi-master setup where all writes go to one of them (and other is set to read-only, and your grants respect read-only) is super operable. If you have your config layer key off of read-only status and you kill existing client connections when you switch from read-write to read-only, it's a pretty small effort to switch masters. I tend to leave the decision to switch masters to humans, and just automate the process. Unless you have bulletproof conflict resolution/reconciliation, automating failover is inviting a conflict that leads to major pain and a large effort to fix. Better to have a few minutes of write downtime while waiting for someone to make the decision (which includes making sure the dead host is truely dead and not just split brained)
juangacovas 5y ago
Sorry, I meant multi-source replication (different masters, different DBs to same slave(s)).

starwatch 5y ago

We ran on this set up for a couple of years. We even migrated from linode to AWS ec2 instances (that was a fun problem!). To start with Galera was okay, but as our traffic picked up things became sketchy. As traffic grew we'd encounter evermore outages - it seemed that some percentage of requests between nodes would be dropped (despite AWS's insistence to the contrary) and the nodes couldn't seem to gracefully recover. With the rapid growth, and tiny team we didn't have enough time to fully get to the bottom of it sadly. The final straw for Galera for us was when we started exploring multi-region replication (us-east-1 and Singapore). The bandwidth between regions was just not high enough to support a cluster. Aurora offered all that and more (though MariaDB to MySQL was a bit fiddly). Since migrating life (and uptime) has improved immeasurably.