4 ms·

We started with a full-stack k8s approach (on GKE); left (switching to plain GCE VMs); then came back much more conservatively, just using GKE for the stateless

by derefr 4y ago

We started with a full-stack k8s approach (on GKE); left (switching to plain GCE VMs); then came back much more conservatively, just using GKE for the stateless business-layer while keeping stateful components on dedicated VMs. Much lower total maintenance burden. (Hard-won bit of experience: k8s + Redis really don't like each-other if Redis 1. is configured to load from disk, and 2. your memory limit for the Redis container is somewhat-tightly bounded. At least from the k8s controller's perspective, Redis apparently uses ~400% of its steady-state memory while reading the AOF tail of an RDB file — getting the container stuck in an OOM-kill loop until you come along and temporarily de-bound its memory.) However, we're considering switching back to k8s for stateful components, with a different approach: allocating single-node node-pools with taints that map 1:1 to each stateful component, effectively making these more like "k8s-managed VMs" than "k8s-managed containers." The point would be to get away from the need to manage the VMs ourselves, giving them over to GKE, while still retaining the assumptions of VM isolation (e.g. not having/needing memory limits, because the single pod is the only tenant of the VM anyway.)

jupp0r 4y ago
Thanks for your insights! One question regarding the approach to hosting stateful components you describe last: what’s the difference between doing this as single node pools vs pod constraints like anti affinity?
- nijave 4y ago
  It would make node pool operations like version upgrades more predictable since you'd know for sure which apps are running a given node pool It can also make monitoring resource usage a little easier since you can just monitor at the node level
paraiuspau 4y ago
I am wondering, did you ever look at tuning MALLOC_ARENA_MAX? This sort of constant consumption of memory is fairly well aligned to the default tuning of MALLOC_ARENA_MAX, which is 8 * nproc. We've just tuned it for a java-based app which was also stuck in OOMkill hell, and this has completely resolved the situation (MALLOC_ARENA_MAX=2).
gizzlon 4y ago
Have you checked out Managed Instance Groups? Used them a wile back, and worked as advertised :) https://cloud.google.com/compute/docs/instance-groups#managed_instance_groups https://cloud.google.com/compute/docs/instance-groups#manage...
roflyear 4y ago
This makes too much sense to me to think people use Kubernetes for anything else. Why must one use Kubernetes for EVERYTHING unless there is some higher-order reason? I use Kubernetes to host something like 6 of our own services, and it excels at that and is fairly simple. Databases and other things use different services.
- adra 4y ago
  We did the reverse of go. We started with purely stateless k8s and got really comfortable with the platform before moving to support stateful loads. For databases, many vendors have dedicated operators which introduces best practice deployments with less operational fuss provisioning the instances. Tooling like Strimzi for Kafka as an example here.
  - AtlasBarfed 4y ago
    This may be the issue with stateful operators: Do you really think they have ALL the operations coded properly for all operational conditions? Stateless is so much easier for operations. It either is running, or not, A/B upgrades, yada yada. Stateful has backups, restores, outages, patching, migrations, corruptions, fixes. For distributed systems it gets even hairier. Do they support your distribution? What if you have a multicloud or blended internal/cloud? Does the operator provide a turnkey restore from backup, and are you testing it? Operators shouldn't be viewed as replacements for stateful operations knowledge, but it's probably what they'll be used for. If you're using a large scale distributed stateful/database system, you need an operations team to support it, or pay for that.
    - unity1001 4y ago
      Stateful has all of those problems anytime, anywhere. Even if you run a single stateful app in a single VM, you will still have all those issues.
      - tpetry 4y ago
        You have a lot less problems in a single VM: There‘s (hopefully) a clear path on how to install software etc. and it is easy to understand. If your distributed PostgreSQL operator written by someone fails you‘re out of luck and it will be really really hard restoring your system. With non-kubernetes non-automated solutions not written by someone you may have some docs saying „Update the replication address and restart“. The way to restore to a running system is much easier in old traditional vm approaches.
MuffinFlavored 4y ago
> Much lower total maintenance burden. Devil's advocate but isn't having to maintain VMs (and then software deployed to those VMs) and k8s YAML/charts/whatever more "maintenance burden" than just one or the other?
- pravus 4y ago
  I guess it would depend on how you managed your deployments/infrastructure but in general I would say no. In my experience stateless services are all managed differently because they typically need fewer resources and can be scaled more easily. Services that require state tend to have a more hands-on approach since they are usually in the critical path for many other services. Deploying a cookie-cutter service is something where k8s accels so it makes sense to use it for those types of workloads.
  - MuffinFlavored 4y ago
    What do you define a server as stateless? As... not needing to talk to a filesystem or a cache server or a database? Just something that takes in API requests (or a cron-like scheduled job) and makes other API calls/does plumbing?
- cogman10 4y ago
  At least in my experience, no. We used managed services for our stateful stuff which significantly eased the operational burden there. Might be a different story if we looked at doing the absolute minimum cost optimization. However, at least for us, the extra cost of managed services is worth the price. The yaml tends to be a "one and done" sort of thing. We touch it MAYBE once every 2 months if that.
zoomzoom 4y ago
At Coherence (withcoherence.com - I'm a cofounder) we generally agree that GKE and other managed k8s are best used for stateless workloads. Rather than move stateful workloads back, leveraging managed services will yield the best results in the long term. In the case of Redis on GCP, something like Memorystore is going to be a better fit than managing a nest of node pools over time (think about version upgrades, resources differences across environments, etc...) However, the complexity of managing the different kinds of configuration across GKE and managed services can be a nightmare. That's a problem we're hoping to help solve, where you define your application and it's dependencies, and we help run it in the right way to leverage managed cloud services across environments without passing that headache on!
- debarshri 4y ago
  Crossplane [1] is great way to create and manage resources across cloud providers, MSPs via kubernetes objects. [1] https://crossplane.io https://crossplane.io
  - aranelsurion 4y ago
    Config Connector [1] is also an option in this space for GCP, it supports many GCP resources and thus far our experience with it has been largely positive. [1] https://cloud.google.com/config-connector/docs/overview https://cloud.google.com/config-connector/docs/overview
Nullabillity 4y ago
> The point would be to get away from the need to manage the VMs ourselves, giving them over to GKE, while still retaining the assumptions of VM isolation (e.g. not having/needing memory limits, because the single pod is the only tenant of the VM anyway.) Isn't this just moving the problem from per-pod resource constraints to per-VM resource constraints?
- scarby2 4y ago
  Yes. They could have just set the memory limit high enough to handle that workload. There's really no different.
- 3np 4y ago
  With the pod, you still have the per-VM/node resource constraints, so it's an additional layer.
foosinn 4y ago
you can set hostAffinities on PersistentVolumes. nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - <hostname> this ensures that your workload will be rescheduled on the matching node
AtNightWeCode 4y ago
Why did you run Redis on K8 in the first place? (One of the reasons we did not move to K8 was the default recommendation to not run Redis, SQL etc on the clusters.)
- derefr 4y ago
  Our Redis use-case started off as ephemeral per-deployment storage for things like rate-limit counters; before evolving into durable per-deployment storage for things like service-metadata discovery. Ephemeral Redis is well-suited to k8s — you can treat it as just a sidecar to your app layer deployment. Durable Redis is not. But sometimes the transition can sneak up on you.
kristaps 4y ago
While I like Kubernetess generally, I agree that the OOM handling is not ideal, feels quite a bit like "exercise left to the reader".
seabrookmx 4y ago
We do something similar with ElasticSearch. We use EKS (a k8s operator) but give each ES node a full k8s node using pod anti-affinities and taints. That way we can just select a sensible disk and instance size on our node pool and not worry about resource request/limit. It's been working very well so far. ES handles node restarts or upgrades pretty gracefully though. I'd imagine for databases or "non-clustered" things you'd have to consider GKE's aggresive upgrade schedule. We use CloudSQL for some databases but our larger ones are still on GCE because we get more control of replication, CDC, and can use tools like proxysql to reduce downtime.
unity1001 4y ago
For stateful stuff, give a try to Google's File Store. It has pretty good performance. As for K8 managed VMs - thats a good idea. Resource management wise, security wise etc.

by derefr 4y ago

jupp0r 4y ago

Thanks for your insights! One question regarding the approach to hosting stateful components you describe last: what’s the difference between doing this as single node pools vs pod constraints like anti affinity?

nijave 4y ago
It would make node pool operations like version upgrades more predictable since you'd know for sure which apps are running a given node pool It can also make monitoring resource usage a little easier since you can just monitor at the node level

paraiuspau 4y ago

I am wondering, did you ever look at tuning MALLOC_ARENA_MAX? This sort of constant consumption of memory is fairly well aligned to the default tuning of MALLOC_ARENA_MAX, which is 8 * nproc. We've just tuned it for a java-based app which was also stuck in OOMkill hell, and this has completely resolved the situation (MALLOC_ARENA_MAX=2).

gizzlon 4y ago

Have you checked out Managed Instance Groups? Used them a wile back, and worked as advertised :) https://cloud.google.com/compute/docs/instance-groups#managed_instance_groups https://cloud.google.com/compute/docs/instance-groups#manage...

roflyear 4y ago

This makes too much sense to me to think people use Kubernetes for anything else. Why must one use Kubernetes for EVERYTHING unless there is some higher-order reason? I use Kubernetes to host something like 6 of our own services, and it excels at that and is fairly simple. Databases and other things use different services.

adra 4y ago
We did the reverse of go. We started with purely stateless k8s and got really comfortable with the platform before moving to support stateful loads. For databases, many vendors have dedicated operators which introduces best practice deployments with less operational fuss provisioning the instances. Tooling like Strimzi for Kafka as an example here.
- AtlasBarfed 4y ago
  This may be the issue with stateful operators: Do you really think they have ALL the operations coded properly for all operational conditions? Stateless is so much easier for operations. It either is running, or not, A/B upgrades, yada yada. Stateful has backups, restores, outages, patching, migrations, corruptions, fixes. For distributed systems it gets even hairier. Do they support your distribution? What if you have a multicloud or blended internal/cloud? Does the operator provide a turnkey restore from backup, and are you testing it? Operators shouldn't be viewed as replacements for stateful operations knowledge, but it's probably what they'll be used for. If you're using a large scale distributed stateful/database system, you need an operations team to support it, or pay for that.
  - unity1001 4y ago
    Stateful has all of those problems anytime, anywhere. Even if you run a single stateful app in a single VM, you will still have all those issues.
    - tpetry 4y ago
      You have a lot less problems in a single VM: There‘s (hopefully) a clear path on how to install software etc. and it is easy to understand. If your distributed PostgreSQL operator written by someone fails you‘re out of luck and it will be really really hard restoring your system. With non-kubernetes non-automated solutions not written by someone you may have some docs saying „Update the replication address and restart“. The way to restore to a running system is much easier in old traditional vm approaches.

MuffinFlavored 4y ago

> Much lower total maintenance burden. Devil's advocate but isn't having to maintain VMs (and then software deployed to those VMs) and k8s YAML/charts/whatever more "maintenance burden" than just one or the other?

pravus 4y ago
I guess it would depend on how you managed your deployments/infrastructure but in general I would say no. In my experience stateless services are all managed differently because they typically need fewer resources and can be scaled more easily. Services that require state tend to have a more hands-on approach since they are usually in the critical path for many other services. Deploying a cookie-cutter service is something where k8s accels so it makes sense to use it for those types of workloads.
- MuffinFlavored 4y ago
  What do you define a server as stateless? As... not needing to talk to a filesystem or a cache server or a database? Just something that takes in API requests (or a cron-like scheduled job) and makes other API calls/does plumbing?
cogman10 4y ago
At least in my experience, no. We used managed services for our stateful stuff which significantly eased the operational burden there. Might be a different story if we looked at doing the absolute minimum cost optimization. However, at least for us, the extra cost of managed services is worth the price. The yaml tends to be a "one and done" sort of thing. We touch it MAYBE once every 2 months if that.

zoomzoom 4y ago

At Coherence (withcoherence.com - I'm a cofounder) we generally agree that GKE and other managed k8s are best used for stateless workloads. Rather than move stateful workloads back, leveraging managed services will yield the best results in the long term. In the case of Redis on GCP, something like Memorystore is going to be a better fit than managing a nest of node pools over time (think about version upgrades, resources differences across environments, etc...) However, the complexity of managing the different kinds of configuration across GKE and managed services can be a nightmare. That's a problem we're hoping to help solve, where you define your application and it's dependencies, and we help run it in the right way to leverage managed cloud services across environments without passing that headache on!

debarshri 4y ago
Crossplane [1] is great way to create and manage resources across cloud providers, MSPs via kubernetes objects. [1] https://crossplane.io https://crossplane.io
- aranelsurion 4y ago
  Config Connector [1] is also an option in this space for GCP, it supports many GCP resources and thus far our experience with it has been largely positive. [1] https://cloud.google.com/config-connector/docs/overview https://cloud.google.com/config-connector/docs/overview

Nullabillity 4y ago

> The point would be to get away from the need to manage the VMs ourselves, giving them over to GKE, while still retaining the assumptions of VM isolation (e.g. not having/needing memory limits, because the single pod is the only tenant of the VM anyway.) Isn't this just moving the problem from per-pod resource constraints to per-VM resource constraints?

scarby2 4y ago
Yes. They could have just set the memory limit high enough to handle that workload. There's really no different.
3np 4y ago
With the pod, you still have the per-VM/node resource constraints, so it's an additional layer.

foosinn 4y ago

you can set hostAffinities on PersistentVolumes. nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - <hostname> this ensures that your workload will be rescheduled on the matching node

AtNightWeCode 4y ago

Why did you run Redis on K8 in the first place? (One of the reasons we did not move to K8 was the default recommendation to not run Redis, SQL etc on the clusters.)

derefr 4y ago
Our Redis use-case started off as ephemeral per-deployment storage for things like rate-limit counters; before evolving into durable per-deployment storage for things like service-metadata discovery. Ephemeral Redis is well-suited to k8s — you can treat it as just a sidecar to your app layer deployment. Durable Redis is not. But sometimes the transition can sneak up on you.

kristaps 4y ago

While I like Kubernetess generally, I agree that the OOM handling is not ideal, feels quite a bit like "exercise left to the reader".

seabrookmx 4y ago

We do something similar with ElasticSearch. We use EKS (a k8s operator) but give each ES node a full k8s node using pod anti-affinities and taints. That way we can just select a sensible disk and instance size on our node pool and not worry about resource request/limit. It's been working very well so far. ES handles node restarts or upgrades pretty gracefully though. I'd imagine for databases or "non-clustered" things you'd have to consider GKE's aggresive upgrade schedule. We use CloudSQL for some databases but our larger ones are still on GCE because we get more control of replication, CDC, and can use tools like proxysql to reduce downtime.

unity1001 4y ago

For stateful stuff, give a try to Google's File Store. It has pretty good performance. As for K8 managed VMs - thats a good idea. Resource management wise, security wise etc.