5 ms·

To sidestep upgrade issues, we're pursuing stateless immutable K8S clusters as much as possible. If we need new K8S, etcd, etc., we'll spin a new cluster and mo

by akeck 8y ago

To sidestep upgrade issues, we're pursuing stateless immutable K8S clusters as much as possible. If we need new K8S, etcd, etc., we'll spin a new cluster and move the apps. Data at rest (prod DBs, prod Posix FS, prod object stores, etc.) is outside the clusters.

SteveNuts 8y ago
Where do you run your persistent apps?
- brianwawok 8y ago
  Not the OP, but I have two kinds of persistent data. 1) Images / files / etc. It all lives in cloud storage ("s3"), outside of K8s 2) RDBMS data. You can just run as hosted sql (say CloudSQL) or a not-in-k8s VM. I have found no compelling reason to move my RDBMS into my k8s cluster.
  - zbentley 8y ago
    That's a bit distressing. Most everywhere I've worked, the infrastructure that matters to the business has fallen into roughly two categories: Category 1: stateless-"ish" workloads. More than 90% of hosts/containers used . . . less than 25% of operations headaches and time. Issues that happen here are solvable with narrow solutions: add caches, scale out, do very targeted, transparent fixes to poorly-performing application code. Category 2: stateful workloads. Less than 10% of hosts/containers. 75% or more of operations headaches and time. Issues that happen here have less visibility, fewer short-term fixes ("just add an index and turn off the bad queries" only works so many times before you're out of low-hanging fruit), and require more expertise to solve in a way that doesn't require the application/clients to change. If k8s and other next-gen technologies are only easing the first category, that makes me sad. It's like we have this sedan (off-the-shelf web technologies) that we have to take off-roading and it falls apart all the time. I don't want a better air conditioning system and more cushions in my seats; I want the vehicle to not break.
    - wereHamster 8y ago
      In k8s you can easily host stateful services. You have persistent disks that you can attach to containers, and you also have StatefulSets if you have a stateful service that you want to have automatically scaled (https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/ https://kubernetes.io/docs/concepts/workloads/controllers/st...). You can use both to run a database (postgres) for example.
      - zaat 8y ago
        I thought that this was the case too, but OP had a link (https://gravitational.com/blog/running-postgresql-on-kubernetes/ https://gravitational.com/blog/running-postgresql-on-kuberne...) to previous post that got me worried.
        outworlder 8y ago
        From TFA > Kubernetes is not aware of the deployment details of Postgres. A naive deployment could lead to complete data loss. That sounds ominous, but is actually a tautology. You have the exact same challenges anywhere else, but since K8s makes some operations so easy to do, you need to be careful. RDBMs are specially tricky because most of them expect a single "master" which holds special status. And it so happens to hold all your data too (as do your replicas, provided they are up to date).
    - thraxil 8y ago
      K8s and similar represent a current view on how systems should be designed and run from the ground up (largely based on the same observations you have made that the stateless workloads can be run with drastically less operational problems). If your architecture lines up with this approach (eg, following the 12-factor approach), there really are a lot of advantages. But no, legacy architected applications are just not going to benefit in the same way. I assume that there must be consultants and "thought leaders" out there who are pushing k8s and containerization as silver bullets and that's unfortunate.
    - mdekkers 8y ago
  - bonesss 8y ago
    > ... not-in-k8s VM One of the cooler innovations we've seen, and I think we're going to see more of, is the ability to take a non-k8s VM and expose it to the cluster as-if it were just another pod. This would let you schedule and expose RDBMs and other specialized servers through kubernetes while keeping them on a standard VM. I think that's the win/win approach to bridge the gap.
    - outworlder 8y ago
      Indeed. But I think that doesn't go far enough: I want to provide K8s with a YAML file and have Kubernetes itself go and create and provision a VM for me. I.e, I want to "kill" Terraform. I don't care about a 'fabric' network (although that could be convenient), just give me an IP that I can reach even it if is external to the cluster. VMs could be just one more resource that can be created or destroyed by k8s, just like network-based storage today. I haven't found this exact use-case implemented yet. If noone else builds it in the coming months I'll probably start doing it.
      - smarterclayton 8y ago
        We did something simple for GCP and CI jobs, when we needed to have VMs in a certain project to launch Kubernetes single node testing - https://github.com/openshift/ci-vm-operator/ https://github.com/openshift/ci-vm-operator/ The base pattern should be pretty easy to modify, although the use case here is very specific.
      - hunter_n 8y ago
        Have you looked at Virtlet or Kubevirt?
bjslade 8y ago
This is the approach we’ve taken with other similar systems too. Things can go so horribly wrong underneath your containers that the concept of having only one cluster in a prod scenario and maintaining it mid-air would be unthinkable. Also agree on keeping state outside. Maybe the relevant tech will be mature sometime soon but we’ve seen orchestration bugs do nasty things to stateless containers that would have been a nightmare if state had been involved.

by akeck 8y ago

SteveNuts 8y ago

Where do you run your persistent apps?

brianwawok 8y ago
Not the OP, but I have two kinds of persistent data. 1) Images / files / etc. It all lives in cloud storage ("s3"), outside of K8s 2) RDBMS data. You can just run as hosted sql (say CloudSQL) or a not-in-k8s VM. I have found no compelling reason to move my RDBMS into my k8s cluster.
- zbentley 8y ago
  That's a bit distressing. Most everywhere I've worked, the infrastructure that matters to the business has fallen into roughly two categories: Category 1: stateless-"ish" workloads. More than 90% of hosts/containers used . . . less than 25% of operations headaches and time. Issues that happen here are solvable with narrow solutions: add caches, scale out, do very targeted, transparent fixes to poorly-performing application code. Category 2: stateful workloads. Less than 10% of hosts/containers. 75% or more of operations headaches and time. Issues that happen here have less visibility, fewer short-term fixes ("just add an index and turn off the bad queries" only works so many times before you're out of low-hanging fruit), and require more expertise to solve in a way that doesn't require the application/clients to change. If k8s and other next-gen technologies are only easing the first category, that makes me sad. It's like we have this sedan (off-the-shelf web technologies) that we have to take off-roading and it falls apart all the time. I don't want a better air conditioning system and more cushions in my seats; I want the vehicle to not break.
  - wereHamster 8y ago
    In k8s you can easily host stateful services. You have persistent disks that you can attach to containers, and you also have StatefulSets if you have a stateful service that you want to have automatically scaled (https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/ https://kubernetes.io/docs/concepts/workloads/controllers/st...). You can use both to run a database (postgres) for example.
    - zaat 8y ago
      I thought that this was the case too, but OP had a link (https://gravitational.com/blog/running-postgresql-on-kubernetes/ https://gravitational.com/blog/running-postgresql-on-kuberne...) to previous post that got me worried.
      - outworlder 8y ago
        From TFA > Kubernetes is not aware of the deployment details of Postgres. A naive deployment could lead to complete data loss. That sounds ominous, but is actually a tautology. You have the exact same challenges anywhere else, but since K8s makes some operations so easy to do, you need to be careful. RDBMs are specially tricky because most of them expect a single "master" which holds special status. And it so happens to hold all your data too (as do your replicas, provided they are up to date).
  - thraxil 8y ago
    K8s and similar represent a current view on how systems should be designed and run from the ground up (largely based on the same observations you have made that the stateless workloads can be run with drastically less operational problems). If your architecture lines up with this approach (eg, following the 12-factor approach), there really are a lot of advantages. But no, legacy architected applications are just not going to benefit in the same way. I assume that there must be consultants and "thought leaders" out there who are pushing k8s and containerization as silver bullets and that's unfortunate.
  - mdekkers 8y ago
- bonesss 8y ago
  > ... not-in-k8s VM One of the cooler innovations we've seen, and I think we're going to see more of, is the ability to take a non-k8s VM and expose it to the cluster as-if it were just another pod. This would let you schedule and expose RDBMs and other specialized servers through kubernetes while keeping them on a standard VM. I think that's the win/win approach to bridge the gap.
  - outworlder 8y ago
    Indeed. But I think that doesn't go far enough: I want to provide K8s with a YAML file and have Kubernetes itself go and create and provision a VM for me. I.e, I want to "kill" Terraform. I don't care about a 'fabric' network (although that could be convenient), just give me an IP that I can reach even it if is external to the cluster. VMs could be just one more resource that can be created or destroyed by k8s, just like network-based storage today. I haven't found this exact use-case implemented yet. If noone else builds it in the coming months I'll probably start doing it.
    - smarterclayton 8y ago
      We did something simple for GCP and CI jobs, when we needed to have VMs in a certain project to launch Kubernetes single node testing - https://github.com/openshift/ci-vm-operator/ https://github.com/openshift/ci-vm-operator/ The base pattern should be pretty easy to modify, although the use case here is very specific.
    - hunter_n 8y ago
      Have you looked at Virtlet or Kubevirt?

bjslade 8y ago

This is the approach we’ve taken with other similar systems too. Things can go so horribly wrong underneath your containers that the concept of having only one cluster in a prod scenario and maintaining it mid-air would be unthinkable. Also agree on keeping state outside. Maybe the relevant tech will be mature sometime soon but we’ve seen orchestration bugs do nasty things to stateless containers that would have been a nightmare if state had been involved.