3 ms·

Putting a 50TB+ database that enables eye watering revenue on k8s is a hard sell for a lot of businesses--especially when they have non-containerized solutions

by nijave 5y ago

Putting a 50TB+ database that enables eye watering revenue on k8s is a hard sell for a lot of businesses--especially when they have non-containerized solutions that work. Simple topologies and NoSQL databases (or databases they can handle replication/partitioning/node failures automatically) are pretty easy to stick in StatefulSets. There's crazy things like this https://blogs.oracle.com/mysql/post/circular-replication-in-mysql https://blogs.oracle.com/mysql/post/circular-replication-in-... that would be fairly difficult to run in a StatefulSet. In addition, the patch and lifecycle cadence of k8s is pretty quick so rebooting SQL databases with 100ks TPS more than 1-2 times a year tends to be problematic

evanelias 5y ago
> There's crazy things like this https://blogs.oracle.com/mysql/post/circular-replication-in-mysql https://blogs.oracle.com/mysql/post/circular-replication-in-... that would be fairly difficult to run in a StatefulSet. This isn't a great example to cite, since traditional circular replication in MySQL is a massive anti-pattern... it's incredibly fragile and pretty much has no valid use-case. That's especially true today when other options like Galera or Group Replication are available. But even 10-15+ years ago, mentioning circular replication was a great way to give a DBA an aneurysm. (Well, I suppose that's arguably a use-case, if you profoundly dislike your company's DBAs...)
- nijave 5y ago
  Probably a bad example, but the sentiment was: encapsulating years of DBA knowledge and complex enterprise architectures in generic k8s abstractions can be incredibly difficult. For instance, you put a RDBMS in a StatefulSet. You figure out how to replicate between pods. Now a piece of hardware fails and a replica needs rebuilt--what does that? So you add an operator into the mix. What happens if bin logs have been purged and a replica can't just "rejoin"? Now you have to figure out how to transfer data between pods (PVs) to bootstrap a new replica. Now you probably need some sort of init container solution to keep the pod running and volume mounted while you're copying in data. Now that you have all that, how do you handle master/primary failures and replica promotion? How do you handle backups? How do you handle restores? Once you've solved all that, how do you performance tune your database? At some point, you're probably going to start looking at kernel tuning which you'll need some additional customizations to enable (probably dedicated node per database pod, then you can either skip k8s and configure the host directly or wire up allowed sysctls through k8s). Or, you can skip all that and require human intervention for everything, but now humans need to wade through the k8s abstractions and fix things. With that route, you break basic k8s functionality like restarting pods since it can take down your database topology
  - evanelias 5y ago
    Oh yes to be clear I definitely agree with the overall sentiment. I just wouldn't ever cite circular replication as an example. Basically saying "it's hard to do [inherently flawed terrible thing] on k8s!" just detracts from the argument :) fwiw my team eventually automated pretty much "everything" for Facebook's database fleet, and it was hundreds of thousands of lines of custom automation code (non-k8s), many years of work for 10+ extremely experienced engineers. In a k8s environment I suspect it would have been even more work. The open source mysql k8s operators are certainly getting better with time, but there's still a lot of stuff they don't handle.
    - nijave 5y ago
      MariaDB uses a closed source operator for their SkySQL product (it's all on Kubernetes). Here's the most recent info I could find http://www.odbms.org/2019/06/on-mariadb-kubernetes-operator-qa-with-saravana-krishnamurthy/ http://www.odbms.org/2019/06/on-mariadb-kubernetes-operator-... (I used to work there)
DeathArrow 5y ago
>Putting a 50TB+ database that enables eye watering revenue on k8s But you can run the software in k8s while using an external data store for data. With microservices you can also use small databases for each and push the data you want to persist like finished transactions and invoices to external systems. No reason to have only one giant database to do everything in. At my last place of work we moved from a monolithic software using a giant database to microservices and many small databases, some of them NoSQL.
- nijave 5y ago
  Parent comment was suggesting you can run databases in StatefulSets on k8s >No reason to have only one giant database to do everything in It's the reality for a lot of "legacy" services. Some of these things have 50-100+ apps all connecting to the same DB with incredibly complex schemas, functions, triggers, etc. It's potentially a multi-year project to undo that. When you do that, you're potentially introducing significantly more network hops. If a business transaction spanned multiple pieces of data in the same database before, you've potentially created a saga spanning multiple microservices now. Even if you do adopt microservices, you'll probably end up consolidating all the data somewhere else for reporting, analytics, data warehousing, etc purposes anyway (so you end up with a giant OLAP DB instead of a giant OLTP one)
koffiezet 5y ago
I have a client who's pretty massive database is pretty damn critical, not only for revenue but also legal reasons. It is managed by a separate team. All applications however run on k8s (well, actually openshift), the main reason being organizational scaling of development teams. With the current setup in place, increasing the amount of dev teams we'd need to support would be pretty painless. We do however run certain smaller (<200gb) databases in-cluster where performance is less of a focus, which, because of the storage abstractions currently in place could otherwise become a potential bottleneck.

by nijave 5y ago

evanelias 5y ago

> There's crazy things like this https://blogs.oracle.com/mysql/post/circular-replication-in-mysql https://blogs.oracle.com/mysql/post/circular-replication-in-... that would be fairly difficult to run in a StatefulSet. This isn't a great example to cite, since traditional circular replication in MySQL is a massive anti-pattern... it's incredibly fragile and pretty much has no valid use-case. That's especially true today when other options like Galera or Group Replication are available. But even 10-15+ years ago, mentioning circular replication was a great way to give a DBA an aneurysm. (Well, I suppose that's arguably a use-case, if you profoundly dislike your company's DBAs...)

nijave 5y ago
Probably a bad example, but the sentiment was: encapsulating years of DBA knowledge and complex enterprise architectures in generic k8s abstractions can be incredibly difficult. For instance, you put a RDBMS in a StatefulSet. You figure out how to replicate between pods. Now a piece of hardware fails and a replica needs rebuilt--what does that? So you add an operator into the mix. What happens if bin logs have been purged and a replica can't just "rejoin"? Now you have to figure out how to transfer data between pods (PVs) to bootstrap a new replica. Now you probably need some sort of init container solution to keep the pod running and volume mounted while you're copying in data. Now that you have all that, how do you handle master/primary failures and replica promotion? How do you handle backups? How do you handle restores? Once you've solved all that, how do you performance tune your database? At some point, you're probably going to start looking at kernel tuning which you'll need some additional customizations to enable (probably dedicated node per database pod, then you can either skip k8s and configure the host directly or wire up allowed sysctls through k8s). Or, you can skip all that and require human intervention for everything, but now humans need to wade through the k8s abstractions and fix things. With that route, you break basic k8s functionality like restarting pods since it can take down your database topology
- evanelias 5y ago
  Oh yes to be clear I definitely agree with the overall sentiment. I just wouldn't ever cite circular replication as an example. Basically saying "it's hard to do [inherently flawed terrible thing] on k8s!" just detracts from the argument :) fwiw my team eventually automated pretty much "everything" for Facebook's database fleet, and it was hundreds of thousands of lines of custom automation code (non-k8s), many years of work for 10+ extremely experienced engineers. In a k8s environment I suspect it would have been even more work. The open source mysql k8s operators are certainly getting better with time, but there's still a lot of stuff they don't handle.
  - nijave 5y ago
    MariaDB uses a closed source operator for their SkySQL product (it's all on Kubernetes). Here's the most recent info I could find http://www.odbms.org/2019/06/on-mariadb-kubernetes-operator-qa-with-saravana-krishnamurthy/ http://www.odbms.org/2019/06/on-mariadb-kubernetes-operator-... (I used to work there)

DeathArrow 5y ago

>Putting a 50TB+ database that enables eye watering revenue on k8s But you can run the software in k8s while using an external data store for data. With microservices you can also use small databases for each and push the data you want to persist like finished transactions and invoices to external systems. No reason to have only one giant database to do everything in. At my last place of work we moved from a monolithic software using a giant database to microservices and many small databases, some of them NoSQL.

nijave 5y ago
Parent comment was suggesting you can run databases in StatefulSets on k8s >No reason to have only one giant database to do everything in It's the reality for a lot of "legacy" services. Some of these things have 50-100+ apps all connecting to the same DB with incredibly complex schemas, functions, triggers, etc. It's potentially a multi-year project to undo that. When you do that, you're potentially introducing significantly more network hops. If a business transaction spanned multiple pieces of data in the same database before, you've potentially created a saga spanning multiple microservices now. Even if you do adopt microservices, you'll probably end up consolidating all the data somewhere else for reporting, analytics, data warehousing, etc purposes anyway (so you end up with a giant OLAP DB instead of a giant OLTP one)

koffiezet 5y ago

I have a client who's pretty massive database is pretty damn critical, not only for revenue but also legal reasons. It is managed by a separate team. All applications however run on k8s (well, actually openshift), the main reason being organizational scaling of development teams. With the current setup in place, increasing the amount of dev teams we'd need to support would be pretty painless. We do however run certain smaller (<200gb) databases in-cluster where performance is less of a focus, which, because of the storage abstractions currently in place could otherwise become a potential bottleneck.