4 ms·

Agreed, we started with Kafka, PostgreSQL and one other service in K8s with StatefulSets. It was painful. We quickly realized that rolling upgrades often got b

Agreed, we started with Kafka, PostgreSQL and one other service in K8s with StatefulSets. It was painful. We quickly realized that rolling upgrades often got blocked. The reason being peristent volume claims sometimes get stuck. And this is just one of few problems. Since we moved databases and stateful services outside Kubernetes, everything got faster and more reliable.

moreira 4y ago
As someone who's been eyeing Kubernetes as a learning experience, that actually is really disheartening to hear. If you're being forced to manage your database outside of Kubernetes, what are you using Kubernetes for? Running your web/worker servers?
- anchochilis 4y ago
  My mid-sized shop uses managed offerings for MySQL and Postgres but we run several stateful workloads, including ElasticSearch and MongoDB, in GKE. I personally haven't experienced the issues the OP is describing. It might be that StatefulSets and PVCs are more stable now then when OP tried them. Of course using managed/hosted K8s makes a big difference.
  - openplatypus 4y ago
    We use managed K8s. And we dropped StatefulSet maybe 4 months ago. There might have been a way to support our use case better. But this was managed service and we couldn't easily tweak every settings. But also, we didn't want to. Our goal of using Kubernetes is to make difficult stuff easy. Not to make easy stuff difficult. And in our skillset and capacity moving stateful services outside K8s was easier, cheaper and more reliable.
- deleted 4y ago
  [deleted]
- openplatypus 4y ago
  Microservices, automations, APIs. K8s is great way to easily deploy and scale services. As we leverage Kafka, a lot of applications fallback on Topics and KTables for storage. Kubernetes with jobs, cron jobs, operators, secrets etc is effectively an Operating System for the cloud. Definitely worth the investment to learn and practice.
- ohthehugemanate 4y ago
  if you're interested in it as a learning experience, this is a great setup. start by putting the stateless parts of your application into k8s, to get the normal flow. then try and move your database in . stateful workloads are tricky and maybe impossible to really do well in k8s, but there are a lot of options that almost get you there, and maybe get you to "good enough." and the learning you will get is fantastic for understanding lots of core k8s capabilities.
- waynesonfire 4y ago
  stop asking questions ... here have some of this koolaid. it's great. it's not complex jedi hand wave and it works for me.
powerhour 4y ago
I don't have a lot of experience with production k8s so I haven't seen too many issues. I have seen PVCs get stuck though. Recently, I was unable to do a rolling restart because I messed up and used zonal disks on a regional cluster. The scheduler would assign pods to nodes in the "wrong" zone and then the disks could never be attached properly. The fix was to add scheduler hints that moved the pods to the correct zone as I should have done in the beginning. (On first deployment, the disk is created on the correct node and thus zone, so it all seemed to work).
- dilyevsky 4y ago
  9/10 times stuckness is a bug in particular csi or node image that you use. Those are fixable. The zonal scheduling thing I believe was fixed a while ago
  - powerhour 4y ago
    Ah, I've been on 1.19 for a while so I'm definitely behind. I'm using GKE regional clusters and as such cannot do zero downtime upgrades to the control plane (they break DNS) so it's risky upgrading.
    - dilyevsky 4y ago
      Hm the topology-aware provisioning should be fixed since at least 1.12/13 and is working well on our self-managed regional clusters. I'd bug GKE support for this... Just curious - how does GKE control plane upgrade break DNS?
      - powerhour 4y ago
        I don't have access to GKE support, unfortunately, so the best I can do is submit product feedback. The cluster's DNS server refuses connections briefly during control plane upgrades or updates. I haven't tried their node local DNS yet (enabling it will cause downtime, natch), maybe it'd help.
nunez 4y ago
PVCs getting stuck in Pending is THE WORST. Sometimes Kubernetes events can tell you why (i.e. no PV matches the claim, PV out of capacity on Kubernetes end), but sometimes it doesn't and you need to find the Pod running the StorageClass to see what went wrong...which can be painful (thank you, NetApp Trident, for making me spend 30 minutes going all over tarnation to figure out why PVCs were stuck only to tell me subtly through a billion lines of logs that the trident-csi couldn't log into Azure) Either way, this is a good example (for me) of how Kubernetes needs some work to be friendlier to operators

moreira 4y ago

As someone who's been eyeing Kubernetes as a learning experience, that actually is really disheartening to hear. If you're being forced to manage your database outside of Kubernetes, what are you using Kubernetes for? Running your web/worker servers?

anchochilis 4y ago
My mid-sized shop uses managed offerings for MySQL and Postgres but we run several stateful workloads, including ElasticSearch and MongoDB, in GKE. I personally haven't experienced the issues the OP is describing. It might be that StatefulSets and PVCs are more stable now then when OP tried them. Of course using managed/hosted K8s makes a big difference.
- openplatypus 4y ago
  We use managed K8s. And we dropped StatefulSet maybe 4 months ago. There might have been a way to support our use case better. But this was managed service and we couldn't easily tweak every settings. But also, we didn't want to. Our goal of using Kubernetes is to make difficult stuff easy. Not to make easy stuff difficult. And in our skillset and capacity moving stateful services outside K8s was easier, cheaper and more reliable.
deleted 4y ago
[deleted]
openplatypus 4y ago
Microservices, automations, APIs. K8s is great way to easily deploy and scale services. As we leverage Kafka, a lot of applications fallback on Topics and KTables for storage. Kubernetes with jobs, cron jobs, operators, secrets etc is effectively an Operating System for the cloud. Definitely worth the investment to learn and practice.
ohthehugemanate 4y ago
if you're interested in it as a learning experience, this is a great setup. start by putting the stateless parts of your application into k8s, to get the normal flow. then try and move your database in . stateful workloads are tricky and maybe impossible to really do well in k8s, but there are a lot of options that almost get you there, and maybe get you to "good enough." and the learning you will get is fantastic for understanding lots of core k8s capabilities.
waynesonfire 4y ago
stop asking questions ... here have some of this koolaid. it's great. it's not complex jedi hand wave and it works for me.

powerhour 4y ago

I don't have a lot of experience with production k8s so I haven't seen too many issues. I have seen PVCs get stuck though. Recently, I was unable to do a rolling restart because I messed up and used zonal disks on a regional cluster. The scheduler would assign pods to nodes in the "wrong" zone and then the disks could never be attached properly. The fix was to add scheduler hints that moved the pods to the correct zone as I should have done in the beginning. (On first deployment, the disk is created on the correct node and thus zone, so it all seemed to work).

dilyevsky 4y ago
9/10 times stuckness is a bug in particular csi or node image that you use. Those are fixable. The zonal scheduling thing I believe was fixed a while ago
- powerhour 4y ago
  Ah, I've been on 1.19 for a while so I'm definitely behind. I'm using GKE regional clusters and as such cannot do zero downtime upgrades to the control plane (they break DNS) so it's risky upgrading.
  - dilyevsky 4y ago
    Hm the topology-aware provisioning should be fixed since at least 1.12/13 and is working well on our self-managed regional clusters. I'd bug GKE support for this... Just curious - how does GKE control plane upgrade break DNS?
    - powerhour 4y ago
      I don't have access to GKE support, unfortunately, so the best I can do is submit product feedback. The cluster's DNS server refuses connections briefly during control plane upgrades or updates. I haven't tried their node local DNS yet (enabling it will cause downtime, natch), maybe it'd help.

nunez 4y ago

PVCs getting stuck in Pending is THE WORST. Sometimes Kubernetes events can tell you why (i.e. no PV matches the claim, PV out of capacity on Kubernetes end), but sometimes it doesn't and you need to find the Pod running the StorageClass to see what went wrong...which can be painful (thank you, NetApp Trident, for making me spend 30 minutes going all over tarnation to figure out why PVCs were stuck only to tell me subtly through a billion lines of logs that the trident-csi couldn't log into Azure) Either way, this is a good example (for me) of how Kubernetes needs some work to be friendlier to operators