4 ms·

People still think stateful things are impossible on k8s but Stateful sets and persistent volumes solves a lot of this. You should be relying on out of the box

People still think stateful things are impossible on k8s but Stateful sets and persistent volumes solves a lot of this. You should be relying on out of the box DB replication to make sure data is available in multiple areas. This is no different on other platforms.

mountainriver 5y ago
Yes you can run DBs on kube now, much of people thinking this isn’t good comes from years back when it wasn’t
- JeremyNT 5y ago
  Does it buy you much though? The big issue with databases is storage, and you need local storage for optimal performance. You can ask k8s for a persistent volume on local storage, but at this point you have to treat that pod exactly the same way you'd treat a snowflake database server, because the local storage is what ultimately matters. You can replicate to other pods with their own persistent volumes, but the replication overhead adds up too both in terms of performance and complexity. I know there's stuff like vitess and crunchy data, which try to abstract this away, but the amount of layers buried here is very high for benefits that seem kind of nebulous compared to "big ol server" mode.
  - mountainriver 5y ago
    pods can share a persistent volume in ReadWriteMany mode
hawk_ 5y ago
Can you elaborate on "multiple areas"? Do you mean each node inside a db "cluster" should run in a different area? And how does one achieve that?
senorsmile 5y ago
Kelsey Hightower just had a good interview where he disagrees with you: https://changelog.com/shipit/44 https://changelog.com/shipit/44 Edit: somewhat* disagrees with you. It's a good listen.
- gabrielgrant 5y ago
  tl;dr?
  - bruth 5y ago
    If you have a low-volume, non-performant, non-critical database, k8s is fine. If you need it to perform and/or need built-in ops (managed backups or replication), use a managed service. k8s _can_ do stateful, but if a managed service exists for this workload, use it. It is not about _can_ I run it on k8s, it is a _should_ question. Is it worth the cumulative effort required to achieve the same degree of quality.
nijave 5y ago
Putting a 50TB+ database that enables eye watering revenue on k8s is a hard sell for a lot of businesses--especially when they have non-containerized solutions that work. Simple topologies and NoSQL databases (or databases they can handle replication/partitioning/node failures automatically) are pretty easy to stick in StatefulSets. There's crazy things like this https://blogs.oracle.com/mysql/post/circular-replication-in-mysql https://blogs.oracle.com/mysql/post/circular-replication-in-... that would be fairly difficult to run in a StatefulSet. In addition, the patch and lifecycle cadence of k8s is pretty quick so rebooting SQL databases with 100ks TPS more than 1-2 times a year tends to be problematic
- evanelias 5y ago
  > There's crazy things like this https://blogs.oracle.com/mysql/post/circular-replication-in-mysql https://blogs.oracle.com/mysql/post/circular-replication-in-... that would be fairly difficult to run in a StatefulSet. This isn't a great example to cite, since traditional circular replication in MySQL is a massive anti-pattern... it's incredibly fragile and pretty much has no valid use-case. That's especially true today when other options like Galera or Group Replication are available. But even 10-15+ years ago, mentioning circular replication was a great way to give a DBA an aneurysm. (Well, I suppose that's arguably a use-case, if you profoundly dislike your company's DBAs...)
  - nijave 5y ago
    Probably a bad example, but the sentiment was: encapsulating years of DBA knowledge and complex enterprise architectures in generic k8s abstractions can be incredibly difficult. For instance, you put a RDBMS in a StatefulSet. You figure out how to replicate between pods. Now a piece of hardware fails and a replica needs rebuilt--what does that? So you add an operator into the mix. What happens if bin logs have been purged and a replica can't just "rejoin"? Now you have to figure out how to transfer data between pods (PVs) to bootstrap a new replica. Now you probably need some sort of init container solution to keep the pod running and volume mounted while you're copying in data. Now that you have all that, how do you handle master/primary failures and replica promotion? How do you handle backups? How do you handle restores? Once you've solved all that, how do you performance tune your database? At some point, you're probably going to start looking at kernel tuning which you'll need some additional customizations to enable (probably dedicated node per database pod, then you can either skip k8s and configure the host directly or wire up allowed sysctls through k8s). Or, you can skip all that and require human intervention for everything, but now humans need to wade through the k8s abstractions and fix things. With that route, you break basic k8s functionality like restarting pods since it can take down your database topology

mountainriver 5y ago

Yes you can run DBs on kube now, much of people thinking this isn’t good comes from years back when it wasn’t

JeremyNT 5y ago
Does it buy you much though? The big issue with databases is storage, and you need local storage for optimal performance. You can ask k8s for a persistent volume on local storage, but at this point you have to treat that pod exactly the same way you'd treat a snowflake database server, because the local storage is what ultimately matters. You can replicate to other pods with their own persistent volumes, but the replication overhead adds up too both in terms of performance and complexity. I know there's stuff like vitess and crunchy data, which try to abstract this away, but the amount of layers buried here is very high for benefits that seem kind of nebulous compared to "big ol server" mode.
- mountainriver 5y ago
  pods can share a persistent volume in ReadWriteMany mode

hawk_ 5y ago

Can you elaborate on "multiple areas"? Do you mean each node inside a db "cluster" should run in a different area? And how does one achieve that?

senorsmile 5y ago

Kelsey Hightower just had a good interview where he disagrees with you: https://changelog.com/shipit/44 https://changelog.com/shipit/44 Edit: somewhat* disagrees with you. It's a good listen.

gabrielgrant 5y ago
tl;dr?
- bruth 5y ago
  If you have a low-volume, non-performant, non-critical database, k8s is fine. If you need it to perform and/or need built-in ops (managed backups or replication), use a managed service. k8s _can_ do stateful, but if a managed service exists for this workload, use it. It is not about _can_ I run it on k8s, it is a _should_ question. Is it worth the cumulative effort required to achieve the same degree of quality.

nijave 5y ago

Putting a 50TB+ database that enables eye watering revenue on k8s is a hard sell for a lot of businesses--especially when they have non-containerized solutions that work. Simple topologies and NoSQL databases (or databases they can handle replication/partitioning/node failures automatically) are pretty easy to stick in StatefulSets. There's crazy things like this https://blogs.oracle.com/mysql/post/circular-replication-in-mysql https://blogs.oracle.com/mysql/post/circular-replication-in-... that would be fairly difficult to run in a StatefulSet. In addition, the patch and lifecycle cadence of k8s is pretty quick so rebooting SQL databases with 100ks TPS more than 1-2 times a year tends to be problematic

evanelias 5y ago
> There's crazy things like this https://blogs.oracle.com/mysql/post/circular-replication-in-mysql https://blogs.oracle.com/mysql/post/circular-replication-in-... that would be fairly difficult to run in a StatefulSet. This isn't a great example to cite, since traditional circular replication in MySQL is a massive anti-pattern... it's incredibly fragile and pretty much has no valid use-case. That's especially true today when other options like Galera or Group Replication are available. But even 10-15+ years ago, mentioning circular replication was a great way to give a DBA an aneurysm. (Well, I suppose that's arguably a use-case, if you profoundly dislike your company's DBAs...)
- nijave 5y ago
  Probably a bad example, but the sentiment was: encapsulating years of DBA knowledge and complex enterprise architectures in generic k8s abstractions can be incredibly difficult. For instance, you put a RDBMS in a StatefulSet. You figure out how to replicate between pods. Now a piece of hardware fails and a replica needs rebuilt--what does that? So you add an operator into the mix. What happens if bin logs have been purged and a replica can't just "rejoin"? Now you have to figure out how to transfer data between pods (PVs) to bootstrap a new replica. Now you probably need some sort of init container solution to keep the pod running and volume mounted while you're copying in data. Now that you have all that, how do you handle master/primary failures and replica promotion? How do you handle backups? How do you handle restores? Once you've solved all that, how do you performance tune your database? At some point, you're probably going to start looking at kernel tuning which you'll need some additional customizations to enable (probably dedicated node per database pod, then you can either skip k8s and configure the host directly or wire up allowed sysctls through k8s). Or, you can skip all that and require human intervention for everything, but now humans need to wade through the k8s abstractions and fix things. With that route, you break basic k8s functionality like restarting pods since it can take down your database topology