5 ms·

Is this an EKS problem or a Kubernetes problem?

user5994461 6y ago
Both. Containers are meant to be ephemeral, they don't play well with permanent storage by design. People have been trying to run databases anyway and hitting roadblock after roadblock. Here's an old rant mentioning broken container filesystems, a bit old from 2016 but storage didn't change much https://thehftguy.com/2016/11/01/docker-in-production-an-history-of-failure/ https://thehftguy.com/2016/11/01/docker-in-production-an-his...
- cbsmith 6y ago
  Unless they are sharded and replicated distributed databases...
- rad_gruchalski 6y ago
  > Both. Containers are meant to be ephemeral, they don't play well with permanent storage by design. People have been trying to run databases anyway and hitting roadblock after roadblock. Really? Can you share details? I'm asking because I know people who run databases in Docker for years and have no issues. Just store data on a volume. > Here's an old rant mentioning broken container filesystems, a bit old from 2016 but storage didn't change much Maybe not, haven't had a storage issue with Docker myself since maybe 2017. That article feels like a big rant but yes, it's ... old.
  - Townley 6y ago
    Docker volumes are great. I've also run databases in docker without issue. But they're different from kubernetes "persistent volumes" and "persistent volume claims" The issues come when you move into high availability (specifically in a Kubernetes context). K8s may decide to move your database to a different node (server) on a whim, and suddenly your data is gone. You have to solve this by having mounted, block-level storage available to your cluster (eg AWS's EBS; the product being discussed) so you can use persistent volumes. That comes with added complexity, and usually the need to incorporate db operators into your stack. As a result, a lot of k8s admins (myself included) choose to pay crazy amounts for RDS and S3 to avoid the complexities of persistent data.
    - jasonjayr 6y ago
      I'm just learning all this; but couldn't you run a DB Cluster persistently on EC2, managed separately from K8s, and VPC it to the K8s cluster? Is there transfer charges involved and/or is the latency to night to consider a setup like this? Is this cost effective to RDS?
      - Townley 6y ago
        If you have a DB cluster running on multiple EC2 instances, you can definitely do that. There are advantages/disadvantages compared to just paying for RDS (eg managing backups, system complexity, and failover) but if you already have that expertise in-house, I can imagine a scale where there might be cost savings. My comment was specifically discussing the challenges of keeping block storage-dependent, stateful apps within k8s (as opposed to using RDS or rolling your own HA cluster on EC2 or elsewhere). Apologies to GP if I incorrectly assumed they were discussing container volumes more broadly.
    - rad_gruchalski 6y ago
      Yes, hence „volume”, not „docker volume”. For k8s, always a PV. rexray for mesos.
- paulfurtado 6y ago
  That article is about broken container filesystems, as in, the root filesystem of the container, but that filesystem is not where you would ever store the data of a database. In kubernetes/eks you would use a PersistentVolumeClaim with an ext4 or xfs filesystem. This would not share any of the overlay filesystem problems and StatefulSets are quite good at managing these. We run over 900 very stateful MySQL database clusters and several hundred zookeeper clusters in kubernetes on ec2 with ebs successfully. Sure, we've hit our share of problems over the years, but storage itself is solid.
  - jrockway 6y ago
    Indeed. The reliability of the system depends on the reliability of the block storage. There are many flaky solutions and many good solutions, and they are unrelated as to whether or not you are using containers.
- parasubvert 6y ago
  Meanwhile arguably the whole reason Kubernetes took off compared to other container orchestrators is support for StatefulSets and Persistent Volumes. Lots of people run their databases on Kubernetes, but it’s best to remember that Pod ephemerality makes it a more complicated orchestrator than them on VMs. That’s said, the market seems to have decided that’s what they want.
  - user5994461 6y ago
    Funny enough, I agree and often hear of statefulset as a pro of kubernetes, yet I've much more rarely seen companies actually using the statefulset let alone migrate their databases to kubernetes. There is a gap between liking a technology and migrating a hundred critical production databases onto it.
    - parasubvert 6y ago
      These things take time. Having sold a stateless container PaaS for many years, the main technical objection is they couldn’t run a database on it, you still needed VMs (even if they were on demand). People caught in tech fads get obsessed with mechanism over outcome.

Both. Containers are meant to be ephemeral, they don't play well with permanent storage by design. People have been trying to run databases anyway and hitting roadblock after roadblock. Here's an old rant mentioning broken container filesystems, a bit old from 2016 but storage didn't change much https://thehftguy.com/2016/11/01/docker-in-production-an-history-of-failure/ https://thehftguy.com/2016/11/01/docker-in-production-an-his...

cbsmith 6y ago
Unless they are sharded and replicated distributed databases...
rad_gruchalski 6y ago
> Both. Containers are meant to be ephemeral, they don't play well with permanent storage by design. People have been trying to run databases anyway and hitting roadblock after roadblock. Really? Can you share details? I'm asking because I know people who run databases in Docker for years and have no issues. Just store data on a volume. > Here's an old rant mentioning broken container filesystems, a bit old from 2016 but storage didn't change much Maybe not, haven't had a storage issue with Docker myself since maybe 2017. That article feels like a big rant but yes, it's ... old.
- Townley 6y ago
  Docker volumes are great. I've also run databases in docker without issue. But they're different from kubernetes "persistent volumes" and "persistent volume claims" The issues come when you move into high availability (specifically in a Kubernetes context). K8s may decide to move your database to a different node (server) on a whim, and suddenly your data is gone. You have to solve this by having mounted, block-level storage available to your cluster (eg AWS's EBS; the product being discussed) so you can use persistent volumes. That comes with added complexity, and usually the need to incorporate db operators into your stack. As a result, a lot of k8s admins (myself included) choose to pay crazy amounts for RDS and S3 to avoid the complexities of persistent data.
  - jasonjayr 6y ago
    I'm just learning all this; but couldn't you run a DB Cluster persistently on EC2, managed separately from K8s, and VPC it to the K8s cluster? Is there transfer charges involved and/or is the latency to night to consider a setup like this? Is this cost effective to RDS?
    - Townley 6y ago
      If you have a DB cluster running on multiple EC2 instances, you can definitely do that. There are advantages/disadvantages compared to just paying for RDS (eg managing backups, system complexity, and failover) but if you already have that expertise in-house, I can imagine a scale where there might be cost savings. My comment was specifically discussing the challenges of keeping block storage-dependent, stateful apps within k8s (as opposed to using RDS or rolling your own HA cluster on EC2 or elsewhere). Apologies to GP if I incorrectly assumed they were discussing container volumes more broadly.
  - rad_gruchalski 6y ago
    Yes, hence „volume”, not „docker volume”. For k8s, always a PV. rexray for mesos.
paulfurtado 6y ago
That article is about broken container filesystems, as in, the root filesystem of the container, but that filesystem is not where you would ever store the data of a database. In kubernetes/eks you would use a PersistentVolumeClaim with an ext4 or xfs filesystem. This would not share any of the overlay filesystem problems and StatefulSets are quite good at managing these. We run over 900 very stateful MySQL database clusters and several hundred zookeeper clusters in kubernetes on ec2 with ebs successfully. Sure, we've hit our share of problems over the years, but storage itself is solid.
- jrockway 6y ago
  Indeed. The reliability of the system depends on the reliability of the block storage. There are many flaky solutions and many good solutions, and they are unrelated as to whether or not you are using containers.
parasubvert 6y ago
Meanwhile arguably the whole reason Kubernetes took off compared to other container orchestrators is support for StatefulSets and Persistent Volumes. Lots of people run their databases on Kubernetes, but it’s best to remember that Pod ephemerality makes it a more complicated orchestrator than them on VMs. That’s said, the market seems to have decided that’s what they want.
- user5994461 6y ago
  Funny enough, I agree and often hear of statefulset as a pro of kubernetes, yet I've much more rarely seen companies actually using the statefulset let alone migrate their databases to kubernetes. There is a gap between liking a technology and migrating a hundred critical production databases onto it.
  - parasubvert 6y ago
    These things take time. Having sold a stateless container PaaS for many years, the main technical objection is they couldn’t run a database on it, you still needed VMs (even if they were on demand). People caught in tech fads get obsessed with mechanism over outcome.