5 ms·

We are currently building a Database-as-a-service platform (tidbcloud.com) using Kubernetes. I have to say it is a love-and-hate story. On the bright side, k8s

We are currently building a Database-as-a-service platform (tidbcloud.com) using Kubernetes. I have to say it is a love-and-hate story. On the bright side, k8s is almost the only option of an abstraction layer on top of different Clouds, for a complex system with tens of components. Database is more than masters and workers, there are so many components you need to take care of. For example, we may need monitoring agents, certificate managers, health checkers, admin proxies, etc. Without k8s, you have to be the owner of a kindergarten. On the other side, k8s IS complicated. It's like an unexplored garden. People just enter it and try to use whatever they see, and cause all kinds of problems. What we met are: * Try to apply the operator pattern to everything, debugging is really painful. Learning curve is steep. * Small services still cost a lot. VPA is not mature enough and many tiny services may be just better off on lambda. * k8s is not really designed for small per-tenant clusters. Managing a fleet of clusters is no easy job, but it is something SaaS companies have to deal with.

nijave 4y ago
Probably the next closest is just plain VMs (and potentially backplane/management layer running on k8s or whatever) But yeah... Even then each cloud has quirks with Kubernetes and there's still quite a few resources just to stand up a cluster. Kubernetes can partially solve the initial provisioning but you generally need the cluster running with nodes before you can use something like CAPI or Crossplane (so you still need Terraform or Pulumi or scripts or whatever) Having worked with a similar system, shared tenancy with tenant per namespace is just as bad but in a different way (if you use the classic operator pattern with 1 operator per cluster, you potentially have a massive blast radius). Then there's security...
- azurezyq 4y ago
  1 operator per cluster is not ideal since most clusters are "stable" and don't need much care. Having plenty of them should be a headache. The operator crash on our side does sound scary. But as a DBaaS system, as long as the blast radius doesn't touch the data plane, it is manageable.
Beltalowda 4y ago
> We are currently building a Database-as-a-service platform (tidbcloud.com) using Kubernetes. I worked for a company that did exactly that (database-as-a-service on k8s); no one in the entire company knew how to run a cluster from scratch. This is a real problem if your developers want to do crazy bizarre advanced stuff like run tests because no one knows how anything fits with anything. At least, I thought it was a real problem as it wrecked any semblance of productivity, but no one else seemed to mind much and thought "it kind-of works on the CI on a good day if the planetary alignments are good" was fine. But hey, VC money so free to piss it all away on a small army of devs being wildly unproductive shrug Also, the reliability of it all was nothing short of embarrassing, and debugging issues was hard because it was all pretty darn complex. Any technology is fantastic if it works, but I think that's the wrong metric. You need to look at how well things will go for you when it doesn't work – for any reason – because you only set your infrastructure up once, but you will "read" (debug) them countless times. I hope it works out better for you, but I always felt that the advantages that k8s gave us could have been implemented from scratch significantly better by a few devs working on it for a few months. The other day I spent about an hour writing a shell script (and then spent a bit of time fixing a bug a few days later) to automate a ~15 minute task. k8s kinda felt like that.
- azurezyq 4y ago
  It actually works better for us. The system is definitely complex, but we still have some ways to debug and develop locally. For example: * You can test different parts of the system individually, via APIs. * With k8s operator model, it's more like a white-box debugging experience. It is not ideal but do-able. * You can have the rest of the system running remotely, but only your own piece locally. As long as the piece can have access to k8s api server, it just works. The best thing k8s offers is repeatability. Scripts are so fragile once the system becomes more complicated. (with monitoring, management agents, etc.) And the product is a distributed database, which itself has so many running parts...
- zinclozenge 4y ago
  > no one in the entire company knew how to run a cluster from scratch How is that even possible? I've worked for 2 dbaas companies that both used k8s and standing up a cluster with a control plane and data plane was as simple as a bash/terraform script. The only thing that was pretty annoying, as I recall, was cert manager because debugging it if you didn't configure it properly was painful, but once it worked it was great. I mean even the non-dbaas companies I worked at didn't have that issue. It sounds like you would have had that problem even if you _didn't_ use k8s.

nijave 4y ago

Probably the next closest is just plain VMs (and potentially backplane/management layer running on k8s or whatever) But yeah... Even then each cloud has quirks with Kubernetes and there's still quite a few resources just to stand up a cluster. Kubernetes can partially solve the initial provisioning but you generally need the cluster running with nodes before you can use something like CAPI or Crossplane (so you still need Terraform or Pulumi or scripts or whatever) Having worked with a similar system, shared tenancy with tenant per namespace is just as bad but in a different way (if you use the classic operator pattern with 1 operator per cluster, you potentially have a massive blast radius). Then there's security...

azurezyq 4y ago
1 operator per cluster is not ideal since most clusters are "stable" and don't need much care. Having plenty of them should be a headache. The operator crash on our side does sound scary. But as a DBaaS system, as long as the blast radius doesn't touch the data plane, it is manageable.

Beltalowda 4y ago

> We are currently building a Database-as-a-service platform (tidbcloud.com) using Kubernetes. I worked for a company that did exactly that (database-as-a-service on k8s); no one in the entire company knew how to run a cluster from scratch. This is a real problem if your developers want to do crazy bizarre advanced stuff like run tests because no one knows how anything fits with anything. At least, I thought it was a real problem as it wrecked any semblance of productivity, but no one else seemed to mind much and thought "it kind-of works on the CI on a good day if the planetary alignments are good" was fine. But hey, VC money so free to piss it all away on a small army of devs being wildly unproductive shrug Also, the reliability of it all was nothing short of embarrassing, and debugging issues was hard because it was all pretty darn complex. Any technology is fantastic if it works, but I think that's the wrong metric. You need to look at how well things will go for you when it doesn't work – for any reason – because you only set your infrastructure up once, but you will "read" (debug) them countless times. I hope it works out better for you, but I always felt that the advantages that k8s gave us could have been implemented from scratch significantly better by a few devs working on it for a few months. The other day I spent about an hour writing a shell script (and then spent a bit of time fixing a bug a few days later) to automate a ~15 minute task. k8s kinda felt like that.

azurezyq 4y ago
It actually works better for us. The system is definitely complex, but we still have some ways to debug and develop locally. For example: * You can test different parts of the system individually, via APIs. * With k8s operator model, it's more like a white-box debugging experience. It is not ideal but do-able. * You can have the rest of the system running remotely, but only your own piece locally. As long as the piece can have access to k8s api server, it just works. The best thing k8s offers is repeatability. Scripts are so fragile once the system becomes more complicated. (with monitoring, management agents, etc.) And the product is a distributed database, which itself has so many running parts...
zinclozenge 4y ago
> no one in the entire company knew how to run a cluster from scratch How is that even possible? I've worked for 2 dbaas companies that both used k8s and standing up a cluster with a control plane and data plane was as simple as a bash/terraform script. The only thing that was pretty annoying, as I recall, was cert manager because debugging it if you didn't configure it properly was painful, but once it worked it was great. I mean even the non-dbaas companies I worked at didn't have that issue. It sounds like you would have had that problem even if you _didn't_ use k8s.