8 ms·

KubeDB – Run production-grade databases easily on Kubernetes

187 points by openmaze 8y ago

lukeqsee 8y ago
Earlier discussion: https://news.ycombinator.com/item?id=18698759 https://news.ycombinator.com/item?id=18698759
SoylentBob 8y ago
Interesting project! Thanks for sharing. How does this compare to other community efforts, e.g. Zalandos Patroni project, aside from supporting more databases than just postgres?
softwaredoug 8y ago
For those terrified of an AWS dominated future, projects like this are crucial. The closer we can get to OSS based push button open source DB cluster in any cloud, the less we need fear AWS will host everything and lock us in to a walled garden of closed source AWS systems.
- twic 8y ago
  This is one place where Cloud Foundry genuinely shines. Part of the architecture of CF is that you have stateful data services provisioned using BOSH, CF's orchestration tool. BOSH can talk to a range of infrastructure providers (AWS, Azure, GCP, VMware). You tell BOSH what to provision using a 'release', and there are releases for, amongst other things, MySQL [1]: https://github.com/cloudfoundry/cf-mysql-release https://github.com/cloudfoundry/cf-mysql-release These releases are used in production by Pivotal, and are actively developed to that end, so they are genuinely production-grade. People have thought carefully about resilience, backups, security, etc. BOSH is a bit awkward, and these releases are tightly coupled to CF, but there's some great work in there.
- deboflo 8y ago
  You are more likely to get locked in with kubernetes than with AWS. It’s easier to migrate out of highly decoupled, well documented systems piece by piece (AWS) than out of monolithic frameworks like k8s.
  - softwaredoug 8y ago
    By “lock in” I should have clarified I meant locked in to a proprietary ecosystem. Certainly being dependent on open source can be problematic if you’re not an active member of the community (or if the “community” is really just one company)
  - derefr 8y ago
    I don't see what the problem is with being locked into a "monolithic framework" as long as you can run your own copy of it. You can take as much time as you like to migrate yourself away from k8s if you don't like it any more (physically migrate your system to your own site; pin the k8s version to prevent API changes; then start changing your code to be less coupled to k8s.) Whereas, if AWS changes and deprecates a feature, you're on their schedule as to how long you have before your service will break.
  - shaklee3 8y ago
    That makes no sense. Kubernetes leveraged aws primitives (elb) if needed, and at its core, it deploys containers. As long as your application runs in a container, you aren't locked in.
    - deboflo 8y ago
      I think we can agree that Kubernetes does far more than schedule containers, even if “at its core” that’s what it does. How many lines of the 2e6 lines of code k8s project are directly related to scheduling containers? Very few. If a scheduler is all that is needed and you want to use any of the 3 different types of load balancers provided by AWS, a simpler architecture might be just to use AWS ECS. 500 lines of declarative Cloudformation or Terraform will do the job.
      - shaklee3 8y ago
        What features are you referring to specifically that lock you in? Sure, it's a large project. But most LOC are around being modular and pluggable, and adhering to standards (OCI, CNI, CSI). I can't think of anything that would be particularly difficult to move out of if needed.
- derefr 8y ago
  I feel like "OSS" is a bit of a misnomer in this case. Your DB cluster, to the degree that it's "production-grade", is partially managed by things like automated upgrade migrations, automatic backups, etc. Essentially, some (centralized!) team associated with the "OSS project" is acting as a devops team for the associated deployments of their project. It's almost as if this team had SSH access to each on-site cluster to ensure their continued smooth operation—but since they don't, they have to do all such maintenance in the form of pre-specifying repair/maintenence strategies, and then building expert-knowledge of when to apply those strategies into the DBMS software itself. But it's still a devops team sitting around doing this—not random contributors. It's a similar thing with e.g. Ubuntu LTS releases. The core distro might be FOSS, but those branches are uniquely the result of a centralized, corporate devops maintainership ensuring that the silent, automatic security and kernel package upgrades go off without a hitch. To be clear, I’m not saying you can’t join that maintainership; what I’m saying is that, unlike with a regular FOSS library or framework, or even a regular piece of FOSS daemon software like Apache, in the case of a DBMS, the software will only continue to run smoothly for as long as that maintainership is around to keep it running smoothly. There’s no such thing as a useful unmaintained DBMS, FOSS or not. And, because of that, the “calculus of TCO” for DBMS projects changes a bit. Unlike regular software, where “proprietary” translates to “higher potential TCO” because of switching costs, in the DBMS case, the “proprietary” vs “open” distinction is nothing next to the “big, healthy maintainership” vs “small, ailing maintainership” distinction. Because, if the DBMS loses all its maintainers? Now you’re stuck maintaining it—at the core level—yourself (and learning how to do so in the process) until such time as you can migrate your data away from it. Personally, for a production-grade DBMS, I’d trust a corporate-backed (or at least sponsored) product over one which is purely a volunteer effort any day.
keypusher 8y ago
While I have completely embraced running stateless services in Docker, I have been hesitant to migrate the database layer to containers. While I have not tested it personally, I have seen numerous reports of performance issues when using volumes. Is this no longer an issue, or was it limited to bind mounts? Do volumes not use the storage driver? Also, I have run into permission issues when using volumes with Docker, which I'm sure was just my own ignorance but it does seem like a cause for confusion and potential error. I have read through the documentation on the linked page, and the quickstart guides for KubeDB seems great for getting up and running, but I do worry about situations like if an automated PG database failover can't reconcile a timeline, there isn't much documentation on failover at all and this could add significant complexity to something that is already a potential nightmare. Anyone care to share their experiences running production databases in k8s?
- cookiecaper 8y ago
  There is no good reason to run non-test database workloads in Kubernetes or Docker. Databases are designed to sit close to the hardware and have a stable, dedicated chunk of resources for a long time, whereas Kubernetes pods are subject to vaporization at any moment. Databases traditionally have fought the operating system to try and maintain enough control to remain performant. Introducing additional layers into this would be dubious at the best of times, but when it's something fundamentally contrary to the application's nature like stateless orchestration, it's pure farce. There could not be an application worse-suited to running in Kubernetes et al than a traditional database. Anyone claiming something that rams this square peg into that round hole is "production ready" is showing that they're an empty husk and shouldn't be trusted near anything important. Note the downvotes already rolling in less than two minutes after I posted this. This subject is a major third rail here. It goes against the agenda of very powerful people and my account has been censured in the past specifically for making this particular argument, that database workloads and Kubernetes don't mix. Keep that in mind when you're asking HN for their experience on this (or any other topic that YC considers critical to the interest of their investments -- they've shown that they're willing to taint the discussion if it gets too dicey).
  - 8y ago
stevenacreman 8y ago
It's good to see a project focussing on production-grade databases on Kubernetes. Particularly the production grade part. There are 33 open source operators for managing databases on Kubernetes. Out of that list only 3 claim to be production ready. Out of 126 Operators that I've looked into the vast majority are abandoned and unfinished. Most state the project status as Alpha in the readme. Kubedb itself has a version number of 0.8.0 for the operator and very low version numbers for the databases. For example version 0.2.0 for Redis. Version numbers can mean anything but they are usually a good indicator of what the project owner thinks the status is. It would be cool to see a break-down of status and expected dates for milestones for Kubedb. For anyone interested in browsing other Operators I keep a table updated half way down this blog post. https://kubedex.com/operators/ https://kubedex.com/operators/ The project statuses come directly from what the authors have stated. Many beta status projects are being used in production.
- manigandham 8y ago
  Part of the problem is that Kubernetes itself is still changing rapidly and already has design-by-committee cracks in the API. It would help if the community took a break from new features and worked on stability first so that Operators and other extensions can finally take off. Some of the things being developed now are so esoteric that it seems to be more about finding the next exciting thing to add than usability.
  - smarterclayton 8y ago
    What are some of the design by committee cracks that you think should be addressed?
  - derefr 8y ago
    > Some of the things being developed now are so esoteric that it seems to be more about finding the next exciting thing to add than usability. Or perhaps it's real ops people with particular arcane needs, each scratching their own itches? K8s is a large FOSS project; and like most large FOSS projects, most PRs are from corporate contributors that wrote the code for their own purposes and then wanted to upstream it to avoid having to maintain a fork.
bearjaws 8y ago
Funny because I was just baffled by the pricing of HA MongoDB (from formerly mlab), it gets way too pricey way too fast. When looking at the hardware being provisioned I realized it wasn't even anything too crazy and could be had for 1/4 the price at Linode. I will definitely be using this in the future.
mosselman 8y ago
Does anyone know of a docker alternative like this? So something like KubeDB that lets me deploy a production-ready postgres db on docker swarm for example?
- cpuguy83 8y ago
  I would not run a database on swarm. It simply does not have the right api's at the cluster level to properly express state requirements. The original swarm design had some of this but it was pulled just before release for more design work... which was never completed. I wrote the only storage support currently in swarm, which is the "mounts" api in your service spec... So, technically you could use swarm to do it, but it will be painful and I don't think any amount of tooling will help until docker includes some support for cluster-aware storage. I would be happy to hear if people have successfully done this, though!
  - mosselman 8y ago
    Thank you for your reply. Do I understand correctly that the biggest issue is the fact that containers won't run on the same node and you'd thus have storage issues? Would these issues be (partially) mitigated if you'd run postgres on a single node?
    - cpuguy83 8y ago
      If you can express your storage requirements via the existing mounts api, then yeah it's all possible. The thing to remember is mounts are implemented only at the node level, so there is no cluster-aware storage controller.
    - keypusher 8y ago
      If you are running multiple copies of postgres on a single node, then you have not significantly improved the resiliency of your database to failure, and it still does not solve the state transition problem. What happens when the primary database fails (or the node dies)? Whether it is on this node or another node, you need to have a replica (sync or async) that you can fail over to, preferably in an automated way. Docker swarm is not equipped to handle these transitions for you, at which point you are just running your database in Docker, with no real benefit over running it on actual hardware or a VM, and with significant added complexity.
rmoriz 8y ago
How does it handle PG updates like for example from PG 9 to PG 10?
- Volundr 8y ago
  Based on my reading of the documentation, it doesn't. So you'd be responsible for taking a backup via pg_dumpall, and restoring it post-upgrade.
  - rmoriz 8y ago
    Thanks for the confirmation. I was not able to find it, either. Strange what use cases are called „production-grade“ nowadays...
    - elsonrodriguez 8y ago
      Anyone that says "Stateful" and "Production Ready" and "Kubernetes" in the same sentence is likely to disappoint you.
geggam 8y ago
Performance tests please ?
an-allen 8y ago
I’ve always been troubled by production-grade handling of state in containers - specifically as it pertains to data backup. This module takes that into account - and defines a “backup k8s object” that will trigger a db dump. But there is still no way to get point in time data recovery/backup that you get from current production-grade managed state providers. Im going to say its production grade if we are using the standards of 10 years ago. Production-grade today, I feel, is a bit more robust.
- DasIch 8y ago
  https://github.com/zalando-incubator/postgres-operator https://github.com/zalando-incubator/postgres-operator supports point in time data recovery just fine and is used in production for 100s of databases at Zalando.
  - pritambarhate 8y ago
    It would be good to know the size and scale of these databases.
    - DasIch 8y ago
      I don’t have actual numbers but I did a quick search and most are a few GiB to tens of GiB, although there are a few hundreds of GiB large. In practice size is not the limiting factor, IOPS are because they all use gp2 EBS volumes. Databases that have huge IOPS requirements are still deployed outside of Kubernetes and run in i3 instances. In that case they still use spilo though, so basically the same system for backups and automatic failover as on Kubernetes. That being said we also have an ElasticSearch operator that is used to deploy ElasticSearch on Kubernetes, there nodes running on i3 instances and the corresponding instance storage is used. Although used in production that’s still very new and sadly not open source.
      - bogomipz 8y ago
        >"In that case they still use spilo though, so basically the same system for backups and automatic failover as on Kubernetes." What is "spilo"? I am not familiar with this term. Thanks.
markbnj 8y ago
I'm wary of the operator model in general, and we haven't had great success using operators to deploy complex stateful services in our clusters. But to be honest we also haven't had great success deploying them using OTS charts from helm stable either. One of our k8s stateful services is a large elasticsearch cluster indexing about 150m events per day, and the chart was forked and heavily modified by us to get it right. I feel that complex stateful services often have enough devils in the details that trying to implement them through an abstraction gets you into trouble. Operators aspire to be a "smart agent" that can translate a CRD resource declaration into a functioning thing, allowing you to implement your data store at an even higher level of abstraction than a helm chart provides. Since in my experience charts are themselves too abstract for this purpose (you either end up forking/modifying or, if the chart actually provides full coverage of the configuration options, creating a whole new hard to comprehend API to the k8s resources you're trying to deploy), I'm not that excited about having a back-end clippie that can do it for us. It's probably fine for simple use cases, and especially those where you often need to create and destroy simple dbs, but imo not yet for large production use cases.
- cyphar 8y ago
  The thing that's always bothered me about the operator model is that you have to reinvent all of the things that Kubernetes does for its own objects (failure handling, redundancy, declarative configuration). At least that's my understanding, from what I've seen.
  - smarterclayton 8y ago
    Arguably all kube is just a pattern library, and the patterns fit reasonably well together. But patterns are situational - you’re always going to have to fill the 20% oc uncovered space with something unique to your use case. Operators are more like frameworks using those patterns. They can reduce time to value, but once you hit the areas where you are fighting the framework you might want to implement those patterns yourself. When we designed stateful sets, it was to make the minimum bar easier to get unique network identity (which is necessary, but not sufficient for most cluster software). And in practice I’ve seen people using sets directly with a thin layer of scripting or helm on top, but I’ve also seen people implementing their own stateful sets because the logic isn’t that hard once pods had the necessary shims. I would probably say that kube is best thought of as an extensible compute pattern framework (you can leverage the lowest level atoms or build layers on top). Kube is probably only successful as long as we keep the 80% easy, reduce the cost to add new patterns (libraries and tools), and decompose the bits that should be replaceable. Most of the “reinventing” problems with operators are problems with kube having weak libraries - each controller and operator is somewhat bespoke. That’s something I expect to see improved this year via the various tooling libraries. But it’s still a work in progress and I regret it took so long.

187 points by openmaze 8y ago

Earlier discussion: https://news.ycombinator.com/item?id=18698759 https://news.ycombinator.com/item?id=18698759

Interesting project! Thanks for sharing. How does this compare to other community efforts, e.g. Zalandos Patroni project, aside from supporting more databases than just postgres?

softwaredoug 8y ago

For those terrified of an AWS dominated future, projects like this are crucial. The closer we can get to OSS based push button open source DB cluster in any cloud, the less we need fear AWS will host everything and lock us in to a walled garden of closed source AWS systems.

twic 8y ago
This is one place where Cloud Foundry genuinely shines. Part of the architecture of CF is that you have stateful data services provisioned using BOSH, CF's orchestration tool. BOSH can talk to a range of infrastructure providers (AWS, Azure, GCP, VMware). You tell BOSH what to provision using a 'release', and there are releases for, amongst other things, MySQL [1]: https://github.com/cloudfoundry/cf-mysql-release https://github.com/cloudfoundry/cf-mysql-release These releases are used in production by Pivotal, and are actively developed to that end, so they are genuinely production-grade. People have thought carefully about resilience, backups, security, etc. BOSH is a bit awkward, and these releases are tightly coupled to CF, but there's some great work in there.
deboflo 8y ago
You are more likely to get locked in with kubernetes than with AWS. It’s easier to migrate out of highly decoupled, well documented systems piece by piece (AWS) than out of monolithic frameworks like k8s.
- softwaredoug 8y ago
  By “lock in” I should have clarified I meant locked in to a proprietary ecosystem. Certainly being dependent on open source can be problematic if you’re not an active member of the community (or if the “community” is really just one company)
- derefr 8y ago
  I don't see what the problem is with being locked into a "monolithic framework" as long as you can run your own copy of it. You can take as much time as you like to migrate yourself away from k8s if you don't like it any more (physically migrate your system to your own site; pin the k8s version to prevent API changes; then start changing your code to be less coupled to k8s.) Whereas, if AWS changes and deprecates a feature, you're on their schedule as to how long you have before your service will break.
- shaklee3 8y ago
  That makes no sense. Kubernetes leveraged aws primitives (elb) if needed, and at its core, it deploys containers. As long as your application runs in a container, you aren't locked in.
  - deboflo 8y ago
    I think we can agree that Kubernetes does far more than schedule containers, even if “at its core” that’s what it does. How many lines of the 2e6 lines of code k8s project are directly related to scheduling containers? Very few. If a scheduler is all that is needed and you want to use any of the 3 different types of load balancers provided by AWS, a simpler architecture might be just to use AWS ECS. 500 lines of declarative Cloudformation or Terraform will do the job.
    - shaklee3 8y ago
      What features are you referring to specifically that lock you in? Sure, it's a large project. But most LOC are around being modular and pluggable, and adhering to standards (OCI, CNI, CSI). I can't think of anything that would be particularly difficult to move out of if needed.
derefr 8y ago
I feel like "OSS" is a bit of a misnomer in this case. Your DB cluster, to the degree that it's "production-grade", is partially managed by things like automated upgrade migrations, automatic backups, etc. Essentially, some (centralized!) team associated with the "OSS project" is acting as a devops team for the associated deployments of their project. It's almost as if this team had SSH access to each on-site cluster to ensure their continued smooth operation—but since they don't, they have to do all such maintenance in the form of pre-specifying repair/maintenence strategies, and then building expert-knowledge of when to apply those strategies into the DBMS software itself. But it's still a devops team sitting around doing this—not random contributors. It's a similar thing with e.g. Ubuntu LTS releases. The core distro might be FOSS, but those branches are uniquely the result of a centralized, corporate devops maintainership ensuring that the silent, automatic security and kernel package upgrades go off without a hitch. To be clear, I’m not saying you can’t join that maintainership; what I’m saying is that, unlike with a regular FOSS library or framework, or even a regular piece of FOSS daemon software like Apache, in the case of a DBMS, the software will only continue to run smoothly for as long as that maintainership is around to keep it running smoothly. There’s no such thing as a useful unmaintained DBMS, FOSS or not. And, because of that, the “calculus of TCO” for DBMS projects changes a bit. Unlike regular software, where “proprietary” translates to “higher potential TCO” because of switching costs, in the DBMS case, the “proprietary” vs “open” distinction is nothing next to the “big, healthy maintainership” vs “small, ailing maintainership” distinction. Because, if the DBMS loses all its maintainers? Now you’re stuck maintaining it—at the core level—yourself (and learning how to do so in the process) until such time as you can migrate your data away from it. Personally, for a production-grade DBMS, I’d trust a corporate-backed (or at least sponsored) product over one which is purely a volunteer effort any day.

keypusher 8y ago

While I have completely embraced running stateless services in Docker, I have been hesitant to migrate the database layer to containers. While I have not tested it personally, I have seen numerous reports of performance issues when using volumes. Is this no longer an issue, or was it limited to bind mounts? Do volumes not use the storage driver? Also, I have run into permission issues when using volumes with Docker, which I'm sure was just my own ignorance but it does seem like a cause for confusion and potential error. I have read through the documentation on the linked page, and the quickstart guides for KubeDB seems great for getting up and running, but I do worry about situations like if an automated PG database failover can't reconcile a timeline, there isn't much documentation on failover at all and this could add significant complexity to something that is already a potential nightmare. Anyone care to share their experiences running production databases in k8s?

cookiecaper 8y ago
There is no good reason to run non-test database workloads in Kubernetes or Docker. Databases are designed to sit close to the hardware and have a stable, dedicated chunk of resources for a long time, whereas Kubernetes pods are subject to vaporization at any moment. Databases traditionally have fought the operating system to try and maintain enough control to remain performant. Introducing additional layers into this would be dubious at the best of times, but when it's something fundamentally contrary to the application's nature like stateless orchestration, it's pure farce. There could not be an application worse-suited to running in Kubernetes et al than a traditional database. Anyone claiming something that rams this square peg into that round hole is "production ready" is showing that they're an empty husk and shouldn't be trusted near anything important. Note the downvotes already rolling in less than two minutes after I posted this. This subject is a major third rail here. It goes against the agenda of very powerful people and my account has been censured in the past specifically for making this particular argument, that database workloads and Kubernetes don't mix. Keep that in mind when you're asking HN for their experience on this (or any other topic that YC considers critical to the interest of their investments -- they've shown that they're willing to taint the discussion if it gets too dicey).
- 8y ago

stevenacreman 8y ago

It's good to see a project focussing on production-grade databases on Kubernetes. Particularly the production grade part. There are 33 open source operators for managing databases on Kubernetes. Out of that list only 3 claim to be production ready. Out of 126 Operators that I've looked into the vast majority are abandoned and unfinished. Most state the project status as Alpha in the readme. Kubedb itself has a version number of 0.8.0 for the operator and very low version numbers for the databases. For example version 0.2.0 for Redis. Version numbers can mean anything but they are usually a good indicator of what the project owner thinks the status is. It would be cool to see a break-down of status and expected dates for milestones for Kubedb. For anyone interested in browsing other Operators I keep a table updated half way down this blog post. https://kubedex.com/operators/ https://kubedex.com/operators/ The project statuses come directly from what the authors have stated. Many beta status projects are being used in production.

manigandham 8y ago
Part of the problem is that Kubernetes itself is still changing rapidly and already has design-by-committee cracks in the API. It would help if the community took a break from new features and worked on stability first so that Operators and other extensions can finally take off. Some of the things being developed now are so esoteric that it seems to be more about finding the next exciting thing to add than usability.
- smarterclayton 8y ago
  What are some of the design by committee cracks that you think should be addressed?
- derefr 8y ago
  > Some of the things being developed now are so esoteric that it seems to be more about finding the next exciting thing to add than usability. Or perhaps it's real ops people with particular arcane needs, each scratching their own itches? K8s is a large FOSS project; and like most large FOSS projects, most PRs are from corporate contributors that wrote the code for their own purposes and then wanted to upstream it to avoid having to maintain a fork.

bearjaws 8y ago

Funny because I was just baffled by the pricing of HA MongoDB (from formerly mlab), it gets way too pricey way too fast. When looking at the hardware being provisioned I realized it wasn't even anything too crazy and could be had for 1/4 the price at Linode. I will definitely be using this in the future.

mosselman 8y ago

Does anyone know of a docker alternative like this? So something like KubeDB that lets me deploy a production-ready postgres db on docker swarm for example?

cpuguy83 8y ago
I would not run a database on swarm. It simply does not have the right api's at the cluster level to properly express state requirements. The original swarm design had some of this but it was pulled just before release for more design work... which was never completed. I wrote the only storage support currently in swarm, which is the "mounts" api in your service spec... So, technically you could use swarm to do it, but it will be painful and I don't think any amount of tooling will help until docker includes some support for cluster-aware storage. I would be happy to hear if people have successfully done this, though!
- mosselman 8y ago
  Thank you for your reply. Do I understand correctly that the biggest issue is the fact that containers won't run on the same node and you'd thus have storage issues? Would these issues be (partially) mitigated if you'd run postgres on a single node?
  - cpuguy83 8y ago
    If you can express your storage requirements via the existing mounts api, then yeah it's all possible. The thing to remember is mounts are implemented only at the node level, so there is no cluster-aware storage controller.
  - keypusher 8y ago
    If you are running multiple copies of postgres on a single node, then you have not significantly improved the resiliency of your database to failure, and it still does not solve the state transition problem. What happens when the primary database fails (or the node dies)? Whether it is on this node or another node, you need to have a replica (sync or async) that you can fail over to, preferably in an automated way. Docker swarm is not equipped to handle these transitions for you, at which point you are just running your database in Docker, with no real benefit over running it on actual hardware or a VM, and with significant added complexity.

rmoriz 8y ago

How does it handle PG updates like for example from PG 9 to PG 10?

Volundr 8y ago
Based on my reading of the documentation, it doesn't. So you'd be responsible for taking a backup via pg_dumpall, and restoring it post-upgrade.
- rmoriz 8y ago
  Thanks for the confirmation. I was not able to find it, either. Strange what use cases are called „production-grade“ nowadays...
  - elsonrodriguez 8y ago
    Anyone that says "Stateful" and "Production Ready" and "Kubernetes" in the same sentence is likely to disappoint you.

geggam 8y ago

Performance tests please ?

an-allen 8y ago

I’ve always been troubled by production-grade handling of state in containers - specifically as it pertains to data backup. This module takes that into account - and defines a “backup k8s object” that will trigger a db dump. But there is still no way to get point in time data recovery/backup that you get from current production-grade managed state providers. Im going to say its production grade if we are using the standards of 10 years ago. Production-grade today, I feel, is a bit more robust.

DasIch 8y ago
https://github.com/zalando-incubator/postgres-operator https://github.com/zalando-incubator/postgres-operator supports point in time data recovery just fine and is used in production for 100s of databases at Zalando.
- pritambarhate 8y ago
  It would be good to know the size and scale of these databases.
  - DasIch 8y ago
    I don’t have actual numbers but I did a quick search and most are a few GiB to tens of GiB, although there are a few hundreds of GiB large. In practice size is not the limiting factor, IOPS are because they all use gp2 EBS volumes. Databases that have huge IOPS requirements are still deployed outside of Kubernetes and run in i3 instances. In that case they still use spilo though, so basically the same system for backups and automatic failover as on Kubernetes. That being said we also have an ElasticSearch operator that is used to deploy ElasticSearch on Kubernetes, there nodes running on i3 instances and the corresponding instance storage is used. Although used in production that’s still very new and sadly not open source.
    - bogomipz 8y ago
      >"In that case they still use spilo though, so basically the same system for backups and automatic failover as on Kubernetes." What is "spilo"? I am not familiar with this term. Thanks.

markbnj 8y ago

I'm wary of the operator model in general, and we haven't had great success using operators to deploy complex stateful services in our clusters. But to be honest we also haven't had great success deploying them using OTS charts from helm stable either. One of our k8s stateful services is a large elasticsearch cluster indexing about 150m events per day, and the chart was forked and heavily modified by us to get it right. I feel that complex stateful services often have enough devils in the details that trying to implement them through an abstraction gets you into trouble. Operators aspire to be a "smart agent" that can translate a CRD resource declaration into a functioning thing, allowing you to implement your data store at an even higher level of abstraction than a helm chart provides. Since in my experience charts are themselves too abstract for this purpose (you either end up forking/modifying or, if the chart actually provides full coverage of the configuration options, creating a whole new hard to comprehend API to the k8s resources you're trying to deploy), I'm not that excited about having a back-end clippie that can do it for us. It's probably fine for simple use cases, and especially those where you often need to create and destroy simple dbs, but imo not yet for large production use cases.

cyphar 8y ago
The thing that's always bothered me about the operator model is that you have to reinvent all of the things that Kubernetes does for its own objects (failure handling, redundancy, declarative configuration). At least that's my understanding, from what I've seen.
- smarterclayton 8y ago
  Arguably all kube is just a pattern library, and the patterns fit reasonably well together. But patterns are situational - you’re always going to have to fill the 20% oc uncovered space with something unique to your use case. Operators are more like frameworks using those patterns. They can reduce time to value, but once you hit the areas where you are fighting the framework you might want to implement those patterns yourself. When we designed stateful sets, it was to make the minimum bar easier to get unique network identity (which is necessary, but not sufficient for most cluster software). And in practice I’ve seen people using sets directly with a thin layer of scripting or helm on top, but I’ve also seen people implementing their own stateful sets because the logic isn’t that hard once pods had the necessary shims. I would probably say that kube is best thought of as an extensible compute pattern framework (you can leverage the lowest level atoms or build layers on top). Kube is probably only successful as long as we keep the 80% easy, reduce the cost to add new patterns (libraries and tools), and decompose the bits that should be replaceable. Most of the “reinventing” problems with operators are problems with kube having weak libraries - each controller and operator is somewhat bespoke. That’s something I expect to see improved this year via the various tooling libraries. But it’s still a work in progress and I regret it took so long.