3 ms·

> k8s is a bad choice if you want to host stateful infrastructure like a DB You cheated again ;) Now we're only running the easy stateless stuff on k8s? I tho

by thunky 4y ago

> k8s is a bad choice if you want to host stateful infrastructure like a DB You cheated again ;) Now we're only running the easy stateless stuff on k8s? I thought k8s gives us simple abstractions that make running stateful services a breeze.

cogman10 4y ago
Why is that cheating? I honestly don’t see why focusing on the strengths of a tech and avoiding the weaknesses is considered “cheating”. The whole point of my initial post was “k8s can be as simple or as complex as you want to make it”. It’s not “cheating” to focus on the parts of k8s that make it simple. It’s not “cheating” to use a tech for it’s strengths and not it’s weaknesses. You could technically use (and some people do use) redis as your primary data store. Is it “cheating” to suggest that’s not a good idea? Is it a bad idea to pull it in to your infrastructure even though it’s not a great at permanent storage? K8s biggest value add is stateless applications. I’ve never suggested otherwise. You can get in the weeds of managing volumes and state if you like, but that adds a fair bit of complexity that, IMO, isn’t worth it vs something like a managed db instance from your cloud provider. I honestly don’t get the “you are cheating by using and benefiting from the simplest features and setup of k8s like you said others should do!”
- vbezhenar 4y ago
  I'm running postgres inside k8s. It's very easy. I don't know why people think that k8s is for stateless.
  - Bayart 4y ago
    I used to think of stateful applications on top of k8s as an anti-pattern until I learned a bit more about some databases uses and changed my entire opinion about what k8s is. If you're coming from the container world, it makes sense to think of orchestration as a way to scale stateless services.
    - cogman10 4y ago
      I'd be glad to hear those usecases. I'm still in the camp of thinking it's not a great idea. I have a hard time thinking coordination between something like postgres and the volume wouldn't be an absolute nightmare to handle. What sorts of volume drivers do you use there? How do you get over the network latency? How does scaling end up working?
      - Bayart 4y ago
        I've not used postgres-on-k8s in any real production workload so my opinion on the matter isn't worth glozing about. I'd rather recommend the blog posts and docs fly.io did on it[1]. They themselves rely on Stolon[2]. AFAIK the secret sauce is postgres' ability for streaming replication[3], ie. pass data across instances as it's ingressed. Add some glue and you've got a cluster. As far as storage goes, k8s supports topology awareness[4] and enables collocating pods and PVs. There are CSI drivers for most file systems you'd encounter in the enterprise[5][6]. > How do you get over the network latency? Ultimately there are incompressible limits you'll hit (CAP theorem and all that). So you do what everybody does in computing : you cheat by exploiting human perception and treating your reads and writes differently[7]. I'm by no means an expert on the subject, but I hope I've answered some of your questions. --- [1]: https://fly.io/blog/globally-distributed-postgres/ https://fly.io/blog/globally-distributed-postgres/ [2]: https://github.com/sorintlab/stolon https://github.com/sorintlab/stolon [3]: https://wiki.postgresql.org/wiki/Streaming_Replication https://wiki.postgresql.org/wiki/Streaming_Replication [4]: https://kubernetes.io/blog/2018/10/11/topology-aware-volume-provisioning-in-kubernetes/ https://kubernetes.io/blog/2018/10/11/topology-aware-volume-... [5]: https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/ https://kubernetes.io/blog/2019/01/15/container-storage-inte... [6]: https://kubernetes-csi.github.io/docs/drivers.html https://kubernetes-csi.github.io/docs/drivers.html [7]: https://fly.io/blog/globally-distributed-postgres/ https://fly.io/blog/globally-distributed-postgres/
      - vbezhenar 4y ago
        You can run database on local volume and not deal with network at all if you don't like it. You can dedicate node for your postrgres instance. Or you can start simple until you hit performance wall and then gradually implement those things. Kubernetes is pretty flexible in that regard.
- thunky 4y ago
  Maybe "cheating" isn't a good way to put it. It just seems like you're narrowing the scope of what k8s is often used for, and then calling it simple and easy. It'd be like someone saying: "<language> can be as simple or complex as you want it do be: it's actually really easy to write hello world. You should steer clear for anything more complex than that". Well sure, but that's not a great way to convey it's overall complexity. But I actually agree with your underlying point. I don't see the benefit or running a db on k8s either. Seems like a lot of extra complexity for little to no benefit.
  - cogman10 4y ago
    > It just seems like you're narrowing the scope of what k8s is often used for, and then calling it simple and easy. I mean, to me, that's what it means to simplify. Even in this post, I see people saying "Hey, it's not actually hard to run postgres in k8s" which, ok, maybe? IDK, seemed like not a great idea to me with the complexity of volume mounts and drivers along with adding extra networking in order to hook it all together. > <language> can be as simple or complex as you want it do be: it's actually really easy to write hello world. You should steer clear for anything more complex than that To some extent, that's exactly what I'm saying. I'd argue you can make things more featureful than "hello world" with stateless pods and you can get a lot of benefit out of those. Just like I'd tell a new Java dev "Hey, java is simple, so long as you don't use features like the LambdaMetaFactory, Reflection, or the annotation processor. Some people use those, but IMO they are mostly off limits" Just because something HAS complex features doesn't mean you should use them (or that using them is a great idea) > But I actually agree with your underlying point. I don't see the benefit or running a db on k8s either. Seems like a lot of extra complexity for little to no benefit. And who knows, maybe this is my (our?) inexperience with k8s shining through. Maybe running postgres on k8s is actually highly beneficial. I've just not been convinced that's better than doing something like amazon's RDS or Aurora. The headache of setting up ceph or rook or whatever just seems like a pretty high burden for a pretty low payback. To me, it's features like deployment rollouts, common logging infrastructure, autoscaling, certificate management/distribution, and recovery that are attractive to k8s and all come mostly out of the box.
    - 4y ago

by thunky 4y ago

cogman10 4y ago

Why is that cheating? I honestly don’t see why focusing on the strengths of a tech and avoiding the weaknesses is considered “cheating”. The whole point of my initial post was “k8s can be as simple or as complex as you want to make it”. It’s not “cheating” to focus on the parts of k8s that make it simple. It’s not “cheating” to use a tech for it’s strengths and not it’s weaknesses. You could technically use (and some people do use) redis as your primary data store. Is it “cheating” to suggest that’s not a good idea? Is it a bad idea to pull it in to your infrastructure even though it’s not a great at permanent storage? K8s biggest value add is stateless applications. I’ve never suggested otherwise. You can get in the weeds of managing volumes and state if you like, but that adds a fair bit of complexity that, IMO, isn’t worth it vs something like a managed db instance from your cloud provider. I honestly don’t get the “you are cheating by using and benefiting from the simplest features and setup of k8s like you said others should do!”

vbezhenar 4y ago
I'm running postgres inside k8s. It's very easy. I don't know why people think that k8s is for stateless.
- Bayart 4y ago
  I used to think of stateful applications on top of k8s as an anti-pattern until I learned a bit more about some databases uses and changed my entire opinion about what k8s is. If you're coming from the container world, it makes sense to think of orchestration as a way to scale stateless services.
  - cogman10 4y ago
    I'd be glad to hear those usecases. I'm still in the camp of thinking it's not a great idea. I have a hard time thinking coordination between something like postgres and the volume wouldn't be an absolute nightmare to handle. What sorts of volume drivers do you use there? How do you get over the network latency? How does scaling end up working?
    - Bayart 4y ago
      I've not used postgres-on-k8s in any real production workload so my opinion on the matter isn't worth glozing about. I'd rather recommend the blog posts and docs fly.io did on it[1]. They themselves rely on Stolon[2]. AFAIK the secret sauce is postgres' ability for streaming replication[3], ie. pass data across instances as it's ingressed. Add some glue and you've got a cluster. As far as storage goes, k8s supports topology awareness[4] and enables collocating pods and PVs. There are CSI drivers for most file systems you'd encounter in the enterprise[5][6]. > How do you get over the network latency? Ultimately there are incompressible limits you'll hit (CAP theorem and all that). So you do what everybody does in computing : you cheat by exploiting human perception and treating your reads and writes differently[7]. I'm by no means an expert on the subject, but I hope I've answered some of your questions. --- [1]: https://fly.io/blog/globally-distributed-postgres/ https://fly.io/blog/globally-distributed-postgres/ [2]: https://github.com/sorintlab/stolon https://github.com/sorintlab/stolon [3]: https://wiki.postgresql.org/wiki/Streaming_Replication https://wiki.postgresql.org/wiki/Streaming_Replication [4]: https://kubernetes.io/blog/2018/10/11/topology-aware-volume-provisioning-in-kubernetes/ https://kubernetes.io/blog/2018/10/11/topology-aware-volume-... [5]: https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/ https://kubernetes.io/blog/2019/01/15/container-storage-inte... [6]: https://kubernetes-csi.github.io/docs/drivers.html https://kubernetes-csi.github.io/docs/drivers.html [7]: https://fly.io/blog/globally-distributed-postgres/ https://fly.io/blog/globally-distributed-postgres/
    - vbezhenar 4y ago
      You can run database on local volume and not deal with network at all if you don't like it. You can dedicate node for your postrgres instance. Or you can start simple until you hit performance wall and then gradually implement those things. Kubernetes is pretty flexible in that regard.
thunky 4y ago
Maybe "cheating" isn't a good way to put it. It just seems like you're narrowing the scope of what k8s is often used for, and then calling it simple and easy. It'd be like someone saying: "<language> can be as simple or complex as you want it do be: it's actually really easy to write hello world. You should steer clear for anything more complex than that". Well sure, but that's not a great way to convey it's overall complexity. But I actually agree with your underlying point. I don't see the benefit or running a db on k8s either. Seems like a lot of extra complexity for little to no benefit.
- cogman10 4y ago
  > It just seems like you're narrowing the scope of what k8s is often used for, and then calling it simple and easy. I mean, to me, that's what it means to simplify. Even in this post, I see people saying "Hey, it's not actually hard to run postgres in k8s" which, ok, maybe? IDK, seemed like not a great idea to me with the complexity of volume mounts and drivers along with adding extra networking in order to hook it all together. > <language> can be as simple or complex as you want it do be: it's actually really easy to write hello world. You should steer clear for anything more complex than that To some extent, that's exactly what I'm saying. I'd argue you can make things more featureful than "hello world" with stateless pods and you can get a lot of benefit out of those. Just like I'd tell a new Java dev "Hey, java is simple, so long as you don't use features like the LambdaMetaFactory, Reflection, or the annotation processor. Some people use those, but IMO they are mostly off limits" Just because something HAS complex features doesn't mean you should use them (or that using them is a great idea) > But I actually agree with your underlying point. I don't see the benefit or running a db on k8s either. Seems like a lot of extra complexity for little to no benefit. And who knows, maybe this is my (our?) inexperience with k8s shining through. Maybe running postgres on k8s is actually highly beneficial. I've just not been convinced that's better than doing something like amazon's RDS or Aurora. The headache of setting up ceph or rook or whatever just seems like a pretty high burden for a pretty low payback. To me, it's features like deployment rollouts, common logging infrastructure, autoscaling, certificate management/distribution, and recovery that are attractive to k8s and all come mostly out of the box.
  - 4y ago