6 ms·

This may be an unpopular opinion, but I’m not a big fan of containers and K8S. If your app needs a container to run properly, it’s already a mess. While what

This may be an unpopular opinion, but I’m not a big fan of containers and K8S. If your app needs a container to run properly, it’s already a mess. While what K8s has done for containers is freaking impressive, to me it does not make a lot of sense unless you run your own bare metal servers. Even then, the complexity it adds may not be worth it. Did I mention that the tech is not mature enough to just run on autopilot and now instead of worrying about the “devops” for your app/service you are playing catch-up with upgrading your K8s cluster? If you’re in the cloud, VMs + autoscalling or fully managed services (eg S3, lambda, etc) make more sense and allow you to focus on your app. Yes there is lock-in. Yes, if not properly arhitected it can be a mess. I wish we would live in a world where people pick simple over complex and think long term vs chasing the latest hotness.

shoo 8y ago
> If you’re in the cloud, VMs + autoscalling or fully managed services (eg S3, lambda, etc) make more sense and allow you to focus on your app. Yes there is lock-in. Yes, if not properly arhitected it can be a mess. I've just rolled off a project (line of business web app + cluster of workers for background job processing), this more or less describes what we have running. Some of the architecture is a bit wrong (it wasn't designed for AWS but shifted there after it had been running for a year or so) but the system works well enough to deliver value to the business. As a developer I hate state, things that aren't properly isolated, ill-defined system boundaries -- but it's not obvious to me what the business case would be to containerise everything.
- threeseed 8y ago
  > it's not obvious to me what the business case would be to containerise everything Containers allow you to move apps trivially between environments and guarantee that they will just work. It allows you to isolate dependencies between apps e.g. Python 2 versus Python 3. It allows you to move apps between cloud providers or between on premise and cloud. With platforms like Kubernetes it allows you to easily scale and self heal when nodes die. And compared to rewriting your app in Lambda which is expensive and complex it is simple to build a container as almost every language has automated tooling.
  - mirceal 8y ago
    Not sure I agree. If you need a container to do the things you mention you’re already in a pretty bad shape. Not to mention that a lot of people don’t understand what a container is.
    - mmt 8y ago
      > Not to mention that a lot of people don’t understand what a container is. That's part of the appeal, the abstraction away of all those pesky, irrelevant details. Developers just want their app to run, as has been mentioned elsewhere in the thread. That desire is understandable. So long as the abstraction isn't too leaky and nothing underneath breaks, there's no downside. It's all upside in terms of human productivity and time to market. Even introduced inefficiency (if any) is unimportant if VC money is fueling the auto-scaling. There's a popular aphorism about premature optimization. Humans are also, in general, far more expensive than machines, especially at scale, and even a mature/traditional company would ignore this at their peril. If problems do eventually crop up with containers or the toolsets around them, chances are, by then, their sheer popularity will ensure the availability of a cadre of experts who can troubleshoot. They may even understand what a container is, even if it's terribly unfashionable to admit, as is the case with operating systems today. Every new technology/tool has some growing pains and is subject to what some consider misuse (due to ignorance). That doesn't necessarily mean it's best to reject it outright.
      - mirceal 8y ago
        Big fan of playing with new tech. In a safe environment where failure is an option. Don’t go betting the farm on shiny things.
        mmt 8y ago
        "Personally" (professionally), neither am I. This kind of conservatism was formed after decades of sysadmin/ops experience. However, having acquired a strong affinity for startups, I also accept that risk (even complete, betting-the-farm risk) is totally OK, so long as that risk is taken in an informed manner. Startups, especially early ones, are a risky proposition from the get-go, and VC money tends to amplify that. The market doesn't much punish a web startup that grew like crazy but lost some user-generated content by using some "NoSQL" database when it first came out. It does, however, punish the one that failed to grow by being too conservative. That's obviously a false dichotomy, but I believe that's essentially the perception that's created, partly by characterizing new technologies as dangerous because they're new and "shiny". Ultimately, I consider mine to be a service profession and an engineering one. As such, if I think a new tech is too undesirable, it's up to me to provide an alternative that actually addresses the original problems (without offloading the burden onto my users/customers).
threeseed 8y ago
This is such nonsense not to mention hilariously hypocritical since you criticise "chasing the latest hotness" but then advocate for a Serverless architecture. Anyway go and try and build a typical application with Lambda/SAM. It is a nightmare of complexity and all you are doing is moving your logic to AWS where you pay 100x the cost of just running it yourself in a container. And the idea that Kubernetes isn't mature is pretty laughable. It's used everyday by Netflix, eBay, Apple, Microsoft, IBM, Lyft, Uber, Square, Google, Pinterest, Stripe, Airbnb, Yahoo, Salesforce etc. And with AWS you have EKS which allows you to run containers in a HA and managed way.
- mirceal 8y ago
  Ha. EKS is a joke. I have written services powered by lambda. As a matter of fact I’ve used AWS/GCP/Azure. I mentioned lambda as an example of a fully managed service - it was not to “endorse” serverless. Also if you look at a container long enough and squint you will see the words serverless emerge. I also had the “pleasure” of working with K8S. If anything, this looks like a pretty good play from Google to get you to eventually run your workloads in GCP.
- etaioinshrdlu 8y ago
  No project with 759 confirmed bugs on Github should be considered mature, in my opinion.
  - outworlder 8y ago
    This is a nonsensical metric and you know it. How many of these are production impacting? For me, that just reflects a lot of activity. Take a look at how many get resolved. I manage a few hundred worker nodes, they are heavily loaded at all times but happily chugging along. My pager is silent and has been for a while now. Last k8s issue we saw was actually our own fault, due to a misconfiguration.
    - etaioinshrdlu 8y ago
      I guess it is a little silly. But it remains a complicated project under heavy development. I'm the kind of guy who enjoys Python 2.7 because it's deprecated, it's done changing :) I try remember the maxim, Happiness is a Boring stack.
outworlder 8y ago
> This may be an unpopular opinion, but I’m not a big fan of containers and K8S. It is unpopular for a reason. Disclaimer: if you can run solely on cloud managed services + serveless, please do that and do not even look at the rest of this message. This is a very nice approach, although there are some things you need to setup before calling victory (deployment pipeline is one). And, as you mentioned, there is vendor lock-in. Now, containers. Look, no-one WANTS containers. Or VMs. Or anything else. We just want to run our stuff. It just so happens that containers are one of the most useful abstractions there are. Unless someone else comes up with a new abstraction, containers it is. Because a container is at the end of the day, a process. Are you against processes? Or against process isolation in general? You cannot lift an existing service and run serveless, you need to modify it. In many cases, it is not practical. In other cases, they need to be an actual server-like application and hold a connection. Lambda doesn't help there. > While what K8s has done for containers is freaking impressive, to me it does not make a lot of sense unless you run your own bare metal servers. One thing has nothing to do with the other. These are different levels of abstraction, there are challenges when running bare metal servers which are not present in cloud environments. Kubernetes can do so much more in a cloud environment (persistent volume claims, etc). Rolling out network attached storage on bare metal servers is a pain. It is also a pain with Openstack, but at least there is a standard interface there. > If you’re in the cloud, VMs + autoscalling or fully managed services (eg S3, lambda, etc) make more sense and allow you to focus on your app. Sorry, I respectfully disagree. I have spent the last two months implementing automation for deploying a cluster on AWS, with auto-scaling, auto-healing, the works, automatically deployed through Jenkins. It is NOT easy, it is not simple, and it is not focusing on my end application, unless you are ignoring all the technical debt you are incurring. And we DO have several k8s clusters, I will be moving that crap to k8s as soon as I can. Let me make a quick list of what you need: Prep: You can use a barebones (ubuntu|redhat|coreos|etc) VM. In which case, provisioning is not complete once the VM is up by the ASG, you need to install the app. If you use an AMI, you now need to build automation to construct these AMIs. Note: if at this stage you are building AMIs by hand, this is a technical debt, which you will have to pay. Alternatively, you can use something like cloud init. If so, see below: * Create the auto-scaling group * Create the launch configuration * If your AMI is not entirely complete, add user data (or equivalent) here * Set the health checks And you are done! Right? No. What about log rotation? Do you have centralized logging? No, tech debt. Go set it up. What about monitoring? Are you using cloudwatch? Prometheus? Go set that up. What about alerting? Not everything requires a VM to be destroyed, you need to set it up. What about upgrades? Are these cattle servers? Then you have to modify your AMI and launch config. Go automate this (tech debt if not) How are you controlling access? Do you have a team? Are they allowed to SSH? Where and how are you storing the keys? How do you invalidate if a key gets compromised? I could go on, but let's keep at this level because the point is to draw a comparison. With K8s, here's what you do: Create a container image. Dockerfile, fancy Jenkins script, some other mechanism, I don't care. Create an image, put it in a registry somewhere. Create a YAML file describing your 'deployment'. It can be a few lines of code if you don't care about most of the stuff. If you need external access, you can create a service, which is another YAML If you don't have an existing HTTPS load balancer, point to the k8s workers (trivial with something like ingress on GKE) And you are done. This automatically gets you: • Self healing • Scheduling among worker nodes. You can control it or let K8s decide • Bin-packing • Logging (centralized logging requires a one-time step, with fluentd or similar, may be handled by cloud providers) • Similarly, monitoring and alerting require a one-time investment in deploying something like Prometheus, after that is done. Getting prometheus to scrape your pods is very easy to do, easier than deploying in a VM by VM basis • Upgrades: deployments handle that for you. Even replica sets before it, you just needed to apply a new YAML with an updated version • There are no SSH keys to mess around. K8s has certificate-based user access control, with an optional RBAC • The SSH equivalent is kubectl exec • Service discovery: you have DNS records for all local services created for you. The cluster will direct you to the correct node. • Scaling is trivial, but most importantly, quick. kubectl scale deployment --replicas=X. It only takes whatever time is required to download the image and run it. You don't have to spin up a whole operating system • Optional: you can have horizontal pod auto-scaling, so your services can scale up and down automatically. It is not perfect, but it can be a game changer. I cannot imagine how we would be running our operation without K8s. Actually I can: version 1.0 of the app was a bunch of VMs, one for each service. It was nightmarish. Now the push, company-wide, is to move everything to K8s. All VMs, all data stores, all of it. And it has absolutely nothing to do with hype, it has everything to do with proven advantages, compared to most of other alternatives. I guess you could also do Mesos. They have a similar concept, only it's not K8s. Note that SOMETHING needs to run the K8s cluster itself. That something is precisely your auto-scaling groups and VM images. It is less painful with a container-optimized OS (like CoreOS or whatever Google uses)
paulddraper 8y ago
> If your app needs a container to run properly, it’s already a mess. > If you’re in the cloud, VMs + autoscalling or fully managed services (eg S3, lambda, etc) make more sense So running with container isolation is a mess, but serverless isolation is admirable?

shoo 8y ago

> If you’re in the cloud, VMs + autoscalling or fully managed services (eg S3, lambda, etc) make more sense and allow you to focus on your app. Yes there is lock-in. Yes, if not properly arhitected it can be a mess. I've just rolled off a project (line of business web app + cluster of workers for background job processing), this more or less describes what we have running. Some of the architecture is a bit wrong (it wasn't designed for AWS but shifted there after it had been running for a year or so) but the system works well enough to deliver value to the business. As a developer I hate state, things that aren't properly isolated, ill-defined system boundaries -- but it's not obvious to me what the business case would be to containerise everything.

threeseed 8y ago
> it's not obvious to me what the business case would be to containerise everything Containers allow you to move apps trivially between environments and guarantee that they will just work. It allows you to isolate dependencies between apps e.g. Python 2 versus Python 3. It allows you to move apps between cloud providers or between on premise and cloud. With platforms like Kubernetes it allows you to easily scale and self heal when nodes die. And compared to rewriting your app in Lambda which is expensive and complex it is simple to build a container as almost every language has automated tooling.
- mirceal 8y ago
  Not sure I agree. If you need a container to do the things you mention you’re already in a pretty bad shape. Not to mention that a lot of people don’t understand what a container is.
  - mmt 8y ago
    > Not to mention that a lot of people don’t understand what a container is. That's part of the appeal, the abstraction away of all those pesky, irrelevant details. Developers just want their app to run, as has been mentioned elsewhere in the thread. That desire is understandable. So long as the abstraction isn't too leaky and nothing underneath breaks, there's no downside. It's all upside in terms of human productivity and time to market. Even introduced inefficiency (if any) is unimportant if VC money is fueling the auto-scaling. There's a popular aphorism about premature optimization. Humans are also, in general, far more expensive than machines, especially at scale, and even a mature/traditional company would ignore this at their peril. If problems do eventually crop up with containers or the toolsets around them, chances are, by then, their sheer popularity will ensure the availability of a cadre of experts who can troubleshoot. They may even understand what a container is, even if it's terribly unfashionable to admit, as is the case with operating systems today. Every new technology/tool has some growing pains and is subject to what some consider misuse (due to ignorance). That doesn't necessarily mean it's best to reject it outright.
    - mirceal 8y ago
      Big fan of playing with new tech. In a safe environment where failure is an option. Don’t go betting the farm on shiny things.
      - mmt 8y ago
        "Personally" (professionally), neither am I. This kind of conservatism was formed after decades of sysadmin/ops experience. However, having acquired a strong affinity for startups, I also accept that risk (even complete, betting-the-farm risk) is totally OK, so long as that risk is taken in an informed manner. Startups, especially early ones, are a risky proposition from the get-go, and VC money tends to amplify that. The market doesn't much punish a web startup that grew like crazy but lost some user-generated content by using some "NoSQL" database when it first came out. It does, however, punish the one that failed to grow by being too conservative. That's obviously a false dichotomy, but I believe that's essentially the perception that's created, partly by characterizing new technologies as dangerous because they're new and "shiny". Ultimately, I consider mine to be a service profession and an engineering one. As such, if I think a new tech is too undesirable, it's up to me to provide an alternative that actually addresses the original problems (without offloading the burden onto my users/customers).

threeseed 8y ago

This is such nonsense not to mention hilariously hypocritical since you criticise "chasing the latest hotness" but then advocate for a Serverless architecture. Anyway go and try and build a typical application with Lambda/SAM. It is a nightmare of complexity and all you are doing is moving your logic to AWS where you pay 100x the cost of just running it yourself in a container. And the idea that Kubernetes isn't mature is pretty laughable. It's used everyday by Netflix, eBay, Apple, Microsoft, IBM, Lyft, Uber, Square, Google, Pinterest, Stripe, Airbnb, Yahoo, Salesforce etc. And with AWS you have EKS which allows you to run containers in a HA and managed way.

mirceal 8y ago
Ha. EKS is a joke. I have written services powered by lambda. As a matter of fact I’ve used AWS/GCP/Azure. I mentioned lambda as an example of a fully managed service - it was not to “endorse” serverless. Also if you look at a container long enough and squint you will see the words serverless emerge. I also had the “pleasure” of working with K8S. If anything, this looks like a pretty good play from Google to get you to eventually run your workloads in GCP.
etaioinshrdlu 8y ago
No project with 759 confirmed bugs on Github should be considered mature, in my opinion.
- outworlder 8y ago
  This is a nonsensical metric and you know it. How many of these are production impacting? For me, that just reflects a lot of activity. Take a look at how many get resolved. I manage a few hundred worker nodes, they are heavily loaded at all times but happily chugging along. My pager is silent and has been for a while now. Last k8s issue we saw was actually our own fault, due to a misconfiguration.
  - etaioinshrdlu 8y ago
    I guess it is a little silly. But it remains a complicated project under heavy development. I'm the kind of guy who enjoys Python 2.7 because it's deprecated, it's done changing :) I try remember the maxim, Happiness is a Boring stack.

outworlder 8y ago

> This may be an unpopular opinion, but I’m not a big fan of containers and K8S. It is unpopular for a reason. Disclaimer: if you can run solely on cloud managed services + serveless, please do that and do not even look at the rest of this message. This is a very nice approach, although there are some things you need to setup before calling victory (deployment pipeline is one). And, as you mentioned, there is vendor lock-in. Now, containers. Look, no-one WANTS containers. Or VMs. Or anything else. We just want to run our stuff. It just so happens that containers are one of the most useful abstractions there are. Unless someone else comes up with a new abstraction, containers it is. Because a container is at the end of the day, a process. Are you against processes? Or against process isolation in general? You cannot lift an existing service and run serveless, you need to modify it. In many cases, it is not practical. In other cases, they need to be an actual server-like application and hold a connection. Lambda doesn't help there. > While what K8s has done for containers is freaking impressive, to me it does not make a lot of sense unless you run your own bare metal servers. One thing has nothing to do with the other. These are different levels of abstraction, there are challenges when running bare metal servers which are not present in cloud environments. Kubernetes can do so much more in a cloud environment (persistent volume claims, etc). Rolling out network attached storage on bare metal servers is a pain. It is also a pain with Openstack, but at least there is a standard interface there. > If you’re in the cloud, VMs + autoscalling or fully managed services (eg S3, lambda, etc) make more sense and allow you to focus on your app. Sorry, I respectfully disagree. I have spent the last two months implementing automation for deploying a cluster on AWS, with auto-scaling, auto-healing, the works, automatically deployed through Jenkins. It is NOT easy, it is not simple, and it is not focusing on my end application, unless you are ignoring all the technical debt you are incurring. And we DO have several k8s clusters, I will be moving that crap to k8s as soon as I can. Let me make a quick list of what you need: Prep: You can use a barebones (ubuntu|redhat|coreos|etc) VM. In which case, provisioning is not complete once the VM is up by the ASG, you need to install the app. If you use an AMI, you now need to build automation to construct these AMIs. Note: if at this stage you are building AMIs by hand, this is a technical debt, which you will have to pay. Alternatively, you can use something like cloud init. If so, see below: * Create the auto-scaling group * Create the launch configuration * If your AMI is not entirely complete, add user data (or equivalent) here * Set the health checks And you are done! Right? No. What about log rotation? Do you have centralized logging? No, tech debt. Go set it up. What about monitoring? Are you using cloudwatch? Prometheus? Go set that up. What about alerting? Not everything requires a VM to be destroyed, you need to set it up. What about upgrades? Are these cattle servers? Then you have to modify your AMI and launch config. Go automate this (tech debt if not) How are you controlling access? Do you have a team? Are they allowed to SSH? Where and how are you storing the keys? How do you invalidate if a key gets compromised? I could go on, but let's keep at this level because the point is to draw a comparison. With K8s, here's what you do: Create a container image. Dockerfile, fancy Jenkins script, some other mechanism, I don't care. Create an image, put it in a registry somewhere. Create a YAML file describing your 'deployment'. It can be a few lines of code if you don't care about most of the stuff. If you need external access, you can create a service, which is another YAML If you don't have an existing HTTPS load balancer, point to the k8s workers (trivial with something like ingress on GKE) And you are done. This automatically gets you: • Self healing • Scheduling among worker nodes. You can control it or let K8s decide • Bin-packing • Logging (centralized logging requires a one-time step, with fluentd or similar, may be handled by cloud providers) • Similarly, monitoring and alerting require a one-time investment in deploying something like Prometheus, after that is done. Getting prometheus to scrape your pods is very easy to do, easier than deploying in a VM by VM basis • Upgrades: deployments handle that for you. Even replica sets before it, you just needed to apply a new YAML with an updated version • There are no SSH keys to mess around. K8s has certificate-based user access control, with an optional RBAC • The SSH equivalent is kubectl exec • Service discovery: you have DNS records for all local services created for you. The cluster will direct you to the correct node. • Scaling is trivial, but most importantly, quick. kubectl scale deployment --replicas=X. It only takes whatever time is required to download the image and run it. You don't have to spin up a whole operating system • Optional: you can have horizontal pod auto-scaling, so your services can scale up and down automatically. It is not perfect, but it can be a game changer. I cannot imagine how we would be running our operation without K8s. Actually I can: version 1.0 of the app was a bunch of VMs, one for each service. It was nightmarish. Now the push, company-wide, is to move everything to K8s. All VMs, all data stores, all of it. And it has absolutely nothing to do with hype, it has everything to do with proven advantages, compared to most of other alternatives. I guess you could also do Mesos. They have a similar concept, only it's not K8s. Note that SOMETHING needs to run the K8s cluster itself. That something is precisely your auto-scaling groups and VM images. It is less painful with a container-optimized OS (like CoreOS or whatever Google uses)

paulddraper 8y ago

> If your app needs a container to run properly, it’s already a mess. > If you’re in the cloud, VMs + autoscalling or fully managed services (eg S3, lambda, etc) make more sense So running with container isolation is a mess, but serverless isolation is admirable?