4 ms·

> We are currently building a Database-as-a-service platform (tidbcloud.com) using Kubernetes. I worked for a company that did exactly that (database-as-a-serv

> We are currently building a Database-as-a-service platform (tidbcloud.com) using Kubernetes. I worked for a company that did exactly that (database-as-a-service on k8s); no one in the entire company knew how to run a cluster from scratch. This is a real problem if your developers want to do crazy bizarre advanced stuff like run tests because no one knows how anything fits with anything. At least, I thought it was a real problem as it wrecked any semblance of productivity, but no one else seemed to mind much and thought "it kind-of works on the CI on a good day if the planetary alignments are good" was fine. But hey, VC money so free to piss it all away on a small army of devs being wildly unproductive shrug Also, the reliability of it all was nothing short of embarrassing, and debugging issues was hard because it was all pretty darn complex. Any technology is fantastic if it works, but I think that's the wrong metric. You need to look at how well things will go for you when it doesn't work – for any reason – because you only set your infrastructure up once, but you will "read" (debug) them countless times. I hope it works out better for you, but I always felt that the advantages that k8s gave us could have been implemented from scratch significantly better by a few devs working on it for a few months. The other day I spent about an hour writing a shell script (and then spent a bit of time fixing a bug a few days later) to automate a ~15 minute task. k8s kinda felt like that.

azurezyq 4y ago
It actually works better for us. The system is definitely complex, but we still have some ways to debug and develop locally. For example: * You can test different parts of the system individually, via APIs. * With k8s operator model, it's more like a white-box debugging experience. It is not ideal but do-able. * You can have the rest of the system running remotely, but only your own piece locally. As long as the piece can have access to k8s api server, it just works. The best thing k8s offers is repeatability. Scripts are so fragile once the system becomes more complicated. (with monitoring, management agents, etc.) And the product is a distributed database, which itself has so many running parts...
zinclozenge 4y ago
> no one in the entire company knew how to run a cluster from scratch How is that even possible? I've worked for 2 dbaas companies that both used k8s and standing up a cluster with a control plane and data plane was as simple as a bash/terraform script. The only thing that was pretty annoying, as I recall, was cert manager because debugging it if you didn't configure it properly was painful, but once it worked it was great. I mean even the non-dbaas companies I worked at didn't have that issue. It sounds like you would have had that problem even if you _didn't_ use k8s.
- Beltalowda 4y ago
  It's actually been an issue everywhere I've seen k8s be deployed. Part of the problem isn't k8s per se but rather "microservices" and that no one really has a good view of everything ties in with everything else, so running a complete product from scratch becomes really hard, and IMO k8s adds to this confusion and complexity. No doubt there would have been issues if k8s wouldn't have been used, but at least that's (often) easier to understand, so it's 'only' "a mess I can understand and fix" rather than "a mess and wtf I have no idea".

azurezyq 4y ago

It actually works better for us. The system is definitely complex, but we still have some ways to debug and develop locally. For example: * You can test different parts of the system individually, via APIs. * With k8s operator model, it's more like a white-box debugging experience. It is not ideal but do-able. * You can have the rest of the system running remotely, but only your own piece locally. As long as the piece can have access to k8s api server, it just works. The best thing k8s offers is repeatability. Scripts are so fragile once the system becomes more complicated. (with monitoring, management agents, etc.) And the product is a distributed database, which itself has so many running parts...

zinclozenge 4y ago

> no one in the entire company knew how to run a cluster from scratch How is that even possible? I've worked for 2 dbaas companies that both used k8s and standing up a cluster with a control plane and data plane was as simple as a bash/terraform script. The only thing that was pretty annoying, as I recall, was cert manager because debugging it if you didn't configure it properly was painful, but once it worked it was great. I mean even the non-dbaas companies I worked at didn't have that issue. It sounds like you would have had that problem even if you _didn't_ use k8s.

Beltalowda 4y ago
It's actually been an issue everywhere I've seen k8s be deployed. Part of the problem isn't k8s per se but rather "microservices" and that no one really has a good view of everything ties in with everything else, so running a complete product from scratch becomes really hard, and IMO k8s adds to this confusion and complexity. No doubt there would have been issues if k8s wouldn't have been used, but at least that's (often) easier to understand, so it's 'only' "a mess I can understand and fix" rather than "a mess and wtf I have no idea".