4 ms·

> (provided, you are using cloud provided k8s). Well, there you cheated ;) K8s is easy if: - you pay for cloud provided k8s - you don't deviate from the hap

> (provided, you are using cloud provided k8s). Well, there you cheated ;) K8s is easy if: - you pay for cloud provided k8s - you don't deviate from the happy paths - you have someone who's going to fix your cluster when you deviate from the happy path Infrastructure is hard. K8s is a black box wrapping such complexity, but the moment something goes wrong with, let's say, DNS, well, you have to know about DNS to fix the problem (and on top of that you have to know how to fix the problem the k8s way)

cogman10 4y ago
> you pay for cloud provided k8s From what I've seen, the cloud provided k8s isn't much more expensive than the vms for the nodes themselves. Amazon's EKS, for example, is $0.10 per hour per cluster on top of the ec2 instance price for the nodes you select. So why wouldn't you do that? > you don't deviate from the happy paths Agreed, but if you are deviating from happy paths, it is, IMO, valid to question if k8s is even the right choice. For example, I think k8s is a bad choice if you want to host stateful infrastructure like a DB. Very large swaths of infrastructure are covered by stateless services. > you have someone who's going to fix your cluster when you deviate from the happy path This is pretty much axiomatic of any infrastructure. If someone screws up your vm, you need someone who's going to fix your vm when you deviate from the parameters given for allocating vms. > Infrastructure is hard. Agreed > K8s is a black box wrapping such complexity, but the moment something goes wrong with, let's say, DNS, well, you have to know about DNS to fix the problem (and on top of that you have to know how to fix the problem the k8s way) When DNS goes wrong anywhere, you have to know how to fix DNS the envoy way, the consul way, the coredns way, the pihole way, the ubuntu way, the freebsd way. The happy path for k8s is really happy. The sad path is no worse than the sad path of any sort of deployment infrastructure. The default bells and whistles of k8s is something that solves way more problems than is created by needing to learn a new system (For example, liveliness and readiness checks, or cert handling). But not only that, because k8s is becoming (rightfully) so popular, it means that issues you run into are almost always a google away. There are a ton of resources and patterns that are globally applicable to pretty much any k8s cluster.
- thunky 4y ago
  > k8s is a bad choice if you want to host stateful infrastructure like a DB You cheated again ;) Now we're only running the easy stateless stuff on k8s? I thought k8s gives us simple abstractions that make running stateful services a breeze.
  - cogman10 4y ago
    Why is that cheating? I honestly don’t see why focusing on the strengths of a tech and avoiding the weaknesses is considered “cheating”. The whole point of my initial post was “k8s can be as simple or as complex as you want to make it”. It’s not “cheating” to focus on the parts of k8s that make it simple. It’s not “cheating” to use a tech for it’s strengths and not it’s weaknesses. You could technically use (and some people do use) redis as your primary data store. Is it “cheating” to suggest that’s not a good idea? Is it a bad idea to pull it in to your infrastructure even though it’s not a great at permanent storage? K8s biggest value add is stateless applications. I’ve never suggested otherwise. You can get in the weeds of managing volumes and state if you like, but that adds a fair bit of complexity that, IMO, isn’t worth it vs something like a managed db instance from your cloud provider. I honestly don’t get the “you are cheating by using and benefiting from the simplest features and setup of k8s like you said others should do!”
    - vbezhenar 4y ago
      I'm running postgres inside k8s. It's very easy. I don't know why people think that k8s is for stateless.
      - Bayart 4y ago
        I used to think of stateful applications on top of k8s as an anti-pattern until I learned a bit more about some databases uses and changed my entire opinion about what k8s is. If you're coming from the container world, it makes sense to think of orchestration as a way to scale stateless services.
dividedbyzero 4y ago
> - you pay for cloud provided k8s At this point, k8s is a cloud abstraction layer that is flexible enough that what's beneath it doesn't have to be a cloud, but in the vast majority of cases is a cloud. The various managed k8s flavors sit in between bare VMs (you control infrastructure) and any serverless offerings (what's infrastructure?), abstracting away a bunch of hard things while still offering a fair amount of flexibility and control, so you can layer other abstractions on top to make it look and function like serverless functions or a batch pipeline runner to your users or automate operations, governance and compliance tasks. The primary value (at least as I see it) that k8s adds lies in its API and the built-in automation it layers on top of the cloud (or non-cloud) below, which I find is a surprisingly good fit for a very wide range of needs. That's what many orgs that run k8s on bare metal are really after: Providing infrastructure users with a single well-known, high-quality API and making use of the wealth of compatible tooling and third-party products written against that API. Once you get to the point where you have a use for a fair amount of k8s features, I've found that home-grown setups tend to approximate a low-quality k8s clone anyway, so running k8s may end up the less painful route in some cases, but bare metal k8s clusters are a lot more rare than cloud-managed ones for a reason. Simpler setups can of course be run with less effort, and in a lot of cases, that's plenty good enough. A single VM with a monolithic app can go a long way, depending on the requirements.
mountainriver 4y ago
K8s very much isn’t a black box, it’s observable on every level, and has no internal APIs.
julienchastang 4y ago
> something goes wrong with, let's say, DNS I ran into exactly this issue for the cert-manager to obtain a LetsEncrypt certificate for our cluster. (The ACME URL was failing inside the pod: no route to host). It took me quite a long time just to figure out it was a DNS related issue because K8s buries technology behind so many layers. How can I make a simple edit to /etc/hosts? Who knows? Ohh and you are not supposed to do it that way anyway. The solution ended up being forcing the cert-manger to run on a node with no DNS issues using such Byzantine terms as "tolerations". Its that bizarre vocabulary where they are obviously trying really hard to abstract something away from you that you would just rather be able to get your hands on. It reminds me of other absurdly complex technologies from the Java world like J2EE and whatnot.
- ralgozino 4y ago
  > How can I make a simple edit to /etc/hosts? Who knows? In case somebody else needs to do this, you actually can: https://kubernetes.io/docs/tasks/network/customize-hosts-file-for-pods/#adding-additional-entries-with-hostaliases https://kubernetes.io/docs/tasks/network/customize-hosts-fil... But as you said, you are "not supposed to". I'd say that if you need to do that, most of the time there's an underlaying issue that needs to be fixed instead and consider editing the hosts file a workaround.

cogman10 4y ago

> you pay for cloud provided k8s From what I've seen, the cloud provided k8s isn't much more expensive than the vms for the nodes themselves. Amazon's EKS, for example, is $0.10 per hour per cluster on top of the ec2 instance price for the nodes you select. So why wouldn't you do that? > you don't deviate from the happy paths Agreed, but if you are deviating from happy paths, it is, IMO, valid to question if k8s is even the right choice. For example, I think k8s is a bad choice if you want to host stateful infrastructure like a DB. Very large swaths of infrastructure are covered by stateless services. > you have someone who's going to fix your cluster when you deviate from the happy path This is pretty much axiomatic of any infrastructure. If someone screws up your vm, you need someone who's going to fix your vm when you deviate from the parameters given for allocating vms. > Infrastructure is hard. Agreed > K8s is a black box wrapping such complexity, but the moment something goes wrong with, let's say, DNS, well, you have to know about DNS to fix the problem (and on top of that you have to know how to fix the problem the k8s way) When DNS goes wrong anywhere, you have to know how to fix DNS the envoy way, the consul way, the coredns way, the pihole way, the ubuntu way, the freebsd way. The happy path for k8s is really happy. The sad path is no worse than the sad path of any sort of deployment infrastructure. The default bells and whistles of k8s is something that solves way more problems than is created by needing to learn a new system (For example, liveliness and readiness checks, or cert handling). But not only that, because k8s is becoming (rightfully) so popular, it means that issues you run into are almost always a google away. There are a ton of resources and patterns that are globally applicable to pretty much any k8s cluster.

thunky 4y ago
> k8s is a bad choice if you want to host stateful infrastructure like a DB You cheated again ;) Now we're only running the easy stateless stuff on k8s? I thought k8s gives us simple abstractions that make running stateful services a breeze.
- cogman10 4y ago
  Why is that cheating? I honestly don’t see why focusing on the strengths of a tech and avoiding the weaknesses is considered “cheating”. The whole point of my initial post was “k8s can be as simple or as complex as you want to make it”. It’s not “cheating” to focus on the parts of k8s that make it simple. It’s not “cheating” to use a tech for it’s strengths and not it’s weaknesses. You could technically use (and some people do use) redis as your primary data store. Is it “cheating” to suggest that’s not a good idea? Is it a bad idea to pull it in to your infrastructure even though it’s not a great at permanent storage? K8s biggest value add is stateless applications. I’ve never suggested otherwise. You can get in the weeds of managing volumes and state if you like, but that adds a fair bit of complexity that, IMO, isn’t worth it vs something like a managed db instance from your cloud provider. I honestly don’t get the “you are cheating by using and benefiting from the simplest features and setup of k8s like you said others should do!”
  - vbezhenar 4y ago
    I'm running postgres inside k8s. It's very easy. I don't know why people think that k8s is for stateless.
    - Bayart 4y ago
      I used to think of stateful applications on top of k8s as an anti-pattern until I learned a bit more about some databases uses and changed my entire opinion about what k8s is. If you're coming from the container world, it makes sense to think of orchestration as a way to scale stateless services.

dividedbyzero 4y ago

> - you pay for cloud provided k8s At this point, k8s is a cloud abstraction layer that is flexible enough that what's beneath it doesn't have to be a cloud, but in the vast majority of cases is a cloud. The various managed k8s flavors sit in between bare VMs (you control infrastructure) and any serverless offerings (what's infrastructure?), abstracting away a bunch of hard things while still offering a fair amount of flexibility and control, so you can layer other abstractions on top to make it look and function like serverless functions or a batch pipeline runner to your users or automate operations, governance and compliance tasks. The primary value (at least as I see it) that k8s adds lies in its API and the built-in automation it layers on top of the cloud (or non-cloud) below, which I find is a surprisingly good fit for a very wide range of needs. That's what many orgs that run k8s on bare metal are really after: Providing infrastructure users with a single well-known, high-quality API and making use of the wealth of compatible tooling and third-party products written against that API. Once you get to the point where you have a use for a fair amount of k8s features, I've found that home-grown setups tend to approximate a low-quality k8s clone anyway, so running k8s may end up the less painful route in some cases, but bare metal k8s clusters are a lot more rare than cloud-managed ones for a reason. Simpler setups can of course be run with less effort, and in a lot of cases, that's plenty good enough. A single VM with a monolithic app can go a long way, depending on the requirements.

mountainriver 4y ago

K8s very much isn’t a black box, it’s observable on every level, and has no internal APIs.

julienchastang 4y ago

> something goes wrong with, let's say, DNS I ran into exactly this issue for the cert-manager to obtain a LetsEncrypt certificate for our cluster. (The ACME URL was failing inside the pod: no route to host). It took me quite a long time just to figure out it was a DNS related issue because K8s buries technology behind so many layers. How can I make a simple edit to /etc/hosts? Who knows? Ohh and you are not supposed to do it that way anyway. The solution ended up being forcing the cert-manger to run on a node with no DNS issues using such Byzantine terms as "tolerations". Its that bizarre vocabulary where they are obviously trying really hard to abstract something away from you that you would just rather be able to get your hands on. It reminds me of other absurdly complex technologies from the Java world like J2EE and whatnot.

ralgozino 4y ago
> How can I make a simple edit to /etc/hosts? Who knows? In case somebody else needs to do this, you actually can: https://kubernetes.io/docs/tasks/network/customize-hosts-file-for-pods/#adding-additional-entries-with-hostaliases https://kubernetes.io/docs/tasks/network/customize-hosts-fil... But as you said, you are "not supposed to". I'd say that if you need to do that, most of the time there's an underlaying issue that needs to be fixed instead and consider editing the hosts file a workaround.