4 ms·

Kinda relevant as the article won’t really help - what would you use for HA MySQL in a non-Kubernetes setting? Getting automated failovers, deployment of new no

Kinda relevant as the article won’t really help - what would you use for HA MySQL in a non-Kubernetes setting? Getting automated failovers, deployment of new nodes, and ~50k databases?

raffraffraff 2y ago
Hahaha... Not galera! Well kinda. Galera looks like it's an amazing solution, and it can be for careful operators who read the docs, learn the new procedures and ensure that they're running workloads that fit well. It can eliminate the usual HA / replication / failover stuff. But it has the ability to act like a foot-machinegun. Outside kubernetes where you don't have an kubernetes operator that can run the database cluster, spin up nodes and pods, manage volumes etc it's difficult to imagine how you might automate that infrastructure in addition to managing the database cluster, without building a lot of it yourself.
smueller1234 2y ago
Was about to say that we managed upwards of 4k servers worth of MySQL databases (as in 4k baremetal servers worth, not 4k small VMs each having a small MySQL) using "Orchestrator" at Booking.com ten years ago. I checked if it was still kicking before writing this and found that the GitHub repo was archived last year. The next Google hit I find is this article from three days ago: https://www.percona.com/blog/orchestrator-for-managing-mysql-high-availability-using-raft/ https://www.percona.com/blog/orchestrator-for-managing-mysql... Please do some additional research into the state of maintenance of that piece of tech before jumping on it. But it certainly did a lot of powerful things for us back in the day. The automatic promotion of followers was key to our deployment.
- b112 2y ago
  That, and patching MySQL in weird ways. I recall something about the filesystem layer being monkeyed with, because "who needs safe writes, it slows things down"
mianos 2y ago
As it involves multiple steps they use candence-workflow to orchestrate it. You could use it in a non k8s environment. Or something like Prefect.io.
kimi 2y ago
Never tried myself, but there is Vitess as well that should do what you ask - not sure about the limitations. https://vitess.io/ https://vitess.io/
cess11 2y ago
Percona MySQL (i.e. Galera), ProxySQL, keepalived, the flavour of orchestration that is most compatible with your production environment and team. Make sure you up ulimit and systemd LimitNOFILE in the MySQL service config. Read docs closely, preferably bring on someone who's spent some time making mistakes in this area to help out, test diligently before putting into production, including load tests.

Kinda relevant as the article won’t really help - what would you use for HA MySQL in a non-Kubernetes setting? Getting automated failovers, deployment of new nodes, and ~50k databases?

raffraffraff 2y ago

Hahaha... Not galera! Well kinda. Galera looks like it's an amazing solution, and it can be for careful operators who read the docs, learn the new procedures and ensure that they're running workloads that fit well. It can eliminate the usual HA / replication / failover stuff. But it has the ability to act like a foot-machinegun. Outside kubernetes where you don't have an kubernetes operator that can run the database cluster, spin up nodes and pods, manage volumes etc it's difficult to imagine how you might automate that infrastructure in addition to managing the database cluster, without building a lot of it yourself.

smueller1234 2y ago

Was about to say that we managed upwards of 4k servers worth of MySQL databases (as in 4k baremetal servers worth, not 4k small VMs each having a small MySQL) using "Orchestrator" at Booking.com ten years ago. I checked if it was still kicking before writing this and found that the GitHub repo was archived last year. The next Google hit I find is this article from three days ago: https://www.percona.com/blog/orchestrator-for-managing-mysql-high-availability-using-raft/ https://www.percona.com/blog/orchestrator-for-managing-mysql... Please do some additional research into the state of maintenance of that piece of tech before jumping on it. But it certainly did a lot of powerful things for us back in the day. The automatic promotion of followers was key to our deployment.

b112 2y ago
That, and patching MySQL in weird ways. I recall something about the filesystem layer being monkeyed with, because "who needs safe writes, it slows things down"

mianos 2y ago

As it involves multiple steps they use candence-workflow to orchestrate it. You could use it in a non k8s environment. Or something like Prefect.io.

kimi 2y ago

Never tried myself, but there is Vitess as well that should do what you ask - not sure about the limitations. https://vitess.io/ https://vitess.io/

cess11 2y ago

Percona MySQL (i.e. Galera), ProxySQL, keepalived, the flavour of orchestration that is most compatible with your production environment and team. Make sure you up ulimit and systemd LimitNOFILE in the MySQL service config. Read docs closely, preferably bring on someone who's spent some time making mistakes in this area to help out, test diligently before putting into production, including load tests.