4 ms·

These are commonly raised concerns, all of which have answers much simpler than "install this giant distributed system". I'll go ahead and answer them since I t

by jen20 6y ago

These are commonly raised concerns, all of which have answers much simpler than "install this giant distributed system". I'll go ahead and answer them since I take the questions to be in good faith... > How do you run an application on a cluster of plain old linux machines? Build a package, install it in an image, run that image in an autoscaling group (or whatever equivalent your cloud of choice offers). > How do you do load balancing? An Elastic Load Balancer (v1 or v2), HAProxy, an F5 - this is deployment environment specific (just like in Kubernetes). > How do you update your app without downtime? Blue-green deployment, or phased rollout. > How do you ensure all your servers are running the same version of dependencies? Build them from a common image. > How do you update those dependencies? Update the Packer template that builds that image. > How do you replicate your environment if you want to add a new server to your cluster? Start the server from the same image. > If your app has microservices how do services discover each other? Consul, or DNS, depending on your appetite. > How do you mount volumes from cloud storage? It's a bit unclear exactly what you mean here, but I'll assume you mean either block devices (just attach them at machine boot, or on startup if they need a claim), or NFS. > How do you update configuration? Either update Consul and have it propagate configuration, or update a configuration package and push it out. > How do you automatically restart failed applications? Systemd restart policy. > How do you monitor if your applications are working? From outside - something like pingdom, and some kind of continuous testing. It's critical that this is measured from the perspective of a user. > How do you make sure the right number of MongoDB replicas are running at all times? Somewhat flippant answer here: the right number of MongoDB servers is zero. More generally, by limiting the size of an autoscaling group. > How do you view your log files remotely? Cloudwatch, Syslog, SSH (depending on requirements). > How do you port-forward from localhost to your Linux server to test your app locally? SSH.

trepatudo 6y ago
Thank you. I use most of this, I've been using it for years and I just don't talk about it because it's hard to argue when people just want to force an idea that k8s is "really the best way of doing things". Also, haproxy is one of the most reliable software I've ever used.
- lsofzz 6y ago
  > Thank you. I use most of this, I've been using it for years and I just don't talk about it because it's hard to argue when people just want to force an idea that k8s is "really the best way of doing things". I wouldn't call it the best way; Rather a good way because Kubernetes does encapsulate the really good bits from scalability, development, security and reliability aspect. It's not a panacea but if you have team bandwidth to run k8s cluster, it's definitely worth a look. <3 HAProxy. It's a solid piece of software tested to the teeth over the years and is great; Here's the thing: You can run it as your preferred ingress controller too :) https://www.haproxy.com/blog/dissecting-the-haproxy-kubernetes-ingress-controller/ https://www.haproxy.com/blog/dissecting-the-haproxy-kubernet...
mvanbaak 6y ago
> Somewhat flippant answer here: the right number of MongoDB servers is zero. AMEN to this one. Very nice answer. It's the way we run infra.
- dlidstrom 6y ago
  What does this mean?
tomerbd 6y ago
I wish I knew to do all this, is there any guide best practice that I can read like a book that includes all above?
hardwaresofton 6y ago
So if you'll indulge me -- this list is exactly why a system like Kubernetes is valuable and why I think personally that it contains a lot of essential complexity. Kubernetes attempts to do all of the above, which is why it's so massive, and I'd argue it's actually less complex than knowing all the tools above -- but it's an equal degree less universally applicable. In this way, it's perfect for the dev who never wants to "do ops", and less so for the dev that already knows ops (or any regular sysadmin/ops person), because they already know all these answers.
- 15155 6y ago
  Devops person here: I already know all of these answers and would still choose Kubernetes over hand-rolling all of this again and again. "Just use Packer, some AMIs, some ASGs, some CF templates, some ELBs, some EC2 instances!" No thanks: I'll Terraform an EKS cluster in 30 lines of HCL and deploy my applications with a Dockerfile and a handful of YAML files.
  - hardwaresofton 6y ago
    > CF Templates Yeah, this is where it kicked in for me. Never mind the fact that all of that is AWS specific and absolutely doesn't help you if you ever move clouds. Great to know all the stuff below, but Kubernetes is a wonderful abstraction layer above that stuff, and it gets better every day. CF could have become Kubernetes -- it was supposed to be, but it just never got the mixture right (and of course is AWS exclusive).
    - jen20 6y ago
      I find the ridiculous false dichotomy between Terraform for Kubernetes and Cloudformation for more basic infrastructure even more ironic given that I am still the eighth most prolific contributor to Terraform _over three years after leaving HashiCorp_.
  - ex_amazon_sde 6y ago
    Both options are questionable.
lazzlazzlazz 6y ago
Thanks for this great list of answers. I was startled to see someone vomiting a list of unresearched questions as if they constituted a rebuttal. "Taking the bait" was the right call. Thanks again.
karakanb 6y ago
Would you mind pointing out some examples that you can achieve these on a bare-bones installation in an automated manner? Like, I mean sharing some real examples that I can install and forget. You can deploy the simplest "hello world" webserver in any language of your choice. >> How do you run an application on a cluster of plain old linux machines? > Build a package, install it in an image, run that image in an autoscaling group (or whatever equivalent your cloud of choice offers). How come it is any different than running the Docker image on Kubernetes? The image build process is the same, delegating running the image to the platform is the same, the only difference at this point is the name of the platform you are running. Even if you were deploying ZIP archives to Elastic Beanstalk, if it doesn't work as expected, you'd have to debug it as an outsider, and you'd still have to know about the technology. I don't see how it is any different than Kubernetes. >> How do you update your app without downtime? > Blue-green deployment, or phased rollout. How exactly? There are gazillion ways of doing them, they are rough concepts, what we need is a reliably working setup that requires as much effort from us as possible, and there are absolutely no standards on how to do them. Are you going to use Ansible? Maybe just SSH into the node and change the symlink? Maybe some other ways? >> How do you replicate your environment if you want to add a new server to your cluster? > Start the server from the same image. How do you do that? You'd either do that manually on AWS console, or build some tooling to achieve that. If you were to do that via the autoscaling options the vendor is providing, then it is no different than Kubernetes: if that doesn't work then you'd have to debug regardless of the platform that is managing the autoscaling. >> If your app has microservices how do services discover each other? > Consul, or DNS, depending on your appetite What is the difference between trying to learn how does Consul handle service discovery vs how does Kubernetes handle it? >> How do you mount volumes from cloud storage? > It's a bit unclear exactly what you mean here, but I'll assume you mean either block devices (just attach them at machine boot, or on startup if they need a claim), or NFS. Would you mind sharing examples that are not vendor-specific and that'd be configurable on a per-service fashion easily, hopefully without writing any code? >> How do you update configuration? > Either update Consul and have it propagate configuration, or update a configuration package and push it out. How is this any better than pushing your changes to Kubernetes? I personally don't know how does Consul work or how to update a configuration package and push it out to somewhere, I don't even know where to push them. In this context, learning them is also not any better than learning how to do them on Kubernetes. >> How do you automatically restart failed applications? > Systemd restart policy. So, this means that you'd need to learn how to utilize Systemd properly in order to be able to start running your application and write the configuration for that somewhere, and also deal with propagating that configuration to all the machines. >> How do you monitor if your applications are working? > From outside - something like pingdom, and some kind of continuous testing. It's critical that this is measured from the perspective of a user. The question was not really that. The tools like pingdom won't help you if an internal dependency of your application starts failing suddenly. You need a standardized solution for gathering various standard metrics from various services of yours, things like request rate, error rate, request durations, as well as defining custom metrics on the application level such as open database connections, latencies on dependencies, and so on. You will definitely need a proper metrics solution for running any serious workload, and in addition to that you'll also want to be able to alert on some of these metrics. There is no standardized solution for these problems, which means you'll need to roll your own. >> How do you view your log files remotely? > Cloudwatch, Syslog, SSH (depending on requirements). The proper alternative to the Kubernetes' solution is Cloudwatch, and even then the simplicity of `kubectl logs <pod_name>` is still better than trying to understand how Cloudwatch works. >> How do you port-forward from localhost to your Linux server to test your app locally? > SSH. This is not a trivial setup. Let's say you have a service A running remotely but it is not exposed, meaning that you cannot reach it from your local machine, and you'd like to be able to use that while developing your service B locally, how would you set this up in an easy way? The points regarding the images are the same points as any Docker image, so it really boils down to the choice and one doesn't have an advantage over the other in this context. What I am trying to say is: there are quite a lot of problems when running any kind of serious workload, and there are thousands of alternative combinations for solving them, and they were solved even before Kubernetes existed; however, there were no standardized way of doing things, and that's what Kubernetes is allowing people to do. There are definitely downsides of Kubernetes, but trying to point specific examples like these don't help as they are just names of individual software that also have learning curve and they all operate differently. I do wish there was a simpler solution, I wish Docker Swarm succeeded as a strong alternative for Kubernetes for simpler cases as it is brilliant working locally, and I wish we didn't have to deal with all these problems, but it is what it is. As of today, I can write a Golang web application, prepare a 10-lines Dockerfile, write ~50 lines of YAML and I am good to go: I can deploy this application on any Kubernetes cluster on any cloud provider and have all these stuff defined above automatically. Do I need to add a Python application alongside: I just write another 20-lines Dockerfile for that application, again ~50 lines of YAML for Kubernetes deployment and bam, that's it. For both of these services I have automated recovery, load balancing, auto-scaling, rolling deployments, stateless deployments, aggregated logging, without writing any code for any tooling.
Longwelwind 6y ago
I feel like your post describes exactly what Kubernetes and container images would bring to your infra. If you were to deploy a solution like you described, you would get something more complex than simply running Kubernetes, except worse. I suspect that you believe your solution would be simpler only because you are more comfortable with those technologies than with k8s. The more I read criticism of k8s, the more I'm persuaded that what people calls "old boring technologies" truly is "technologies I'm comfortable with". On top of that, you'd need to separately document everything you do on your infra. The advantage of Docker images over AMI is that you have a file that describes how the image was built. With an AMI, you would need to hope that the guy who created the AMI documented it somewhere (or hope that he has not quit). Same goes for k8s, where configurations are committed into your repository. At the end of the day, k8s stays a tool that you should use only when needed (and also if you have the capabilities of using it), but I think you shouldn't discard it simply because you are capable of producing the same result by other means. You get a lot more by using k8s, in my opinion.
- jen20 6y ago
  > The advantage of Docker images over AMI is that you have a file that describes how the image was built. https://packer.io https://packer.io - how is this even a discussion? > the more I'm persuaded that what people calls "old boring technologies" truly is "technologies I'm comfortable with". My infrastructure runs in Nomad on bare metal. I am by no means opposed to “progress”, I just don’t think Kubernetes is the be-all-and-end-all of infrastructure and would like to have a less hysterical debate about it than the parent to my original post presented.
lenkite 6y ago
I loved this list of succinct answers.
sciurus 6y ago
Great answers. To be fair though, that's not "Plain old linux" like zaptheimpaler suggested was somehow possible. That's linux plus AWS managed services plus software from Hashicorp. Which is a great stack to be on, but has its own complexities and tradeoffs.

by jen20 6y ago

trepatudo 6y ago

Thank you. I use most of this, I've been using it for years and I just don't talk about it because it's hard to argue when people just want to force an idea that k8s is "really the best way of doing things". Also, haproxy is one of the most reliable software I've ever used.

lsofzz 6y ago
> Thank you. I use most of this, I've been using it for years and I just don't talk about it because it's hard to argue when people just want to force an idea that k8s is "really the best way of doing things". I wouldn't call it the best way; Rather a good way because Kubernetes does encapsulate the really good bits from scalability, development, security and reliability aspect. It's not a panacea but if you have team bandwidth to run k8s cluster, it's definitely worth a look. <3 HAProxy. It's a solid piece of software tested to the teeth over the years and is great; Here's the thing: You can run it as your preferred ingress controller too :) https://www.haproxy.com/blog/dissecting-the-haproxy-kubernetes-ingress-controller/ https://www.haproxy.com/blog/dissecting-the-haproxy-kubernet...

mvanbaak 6y ago

> Somewhat flippant answer here: the right number of MongoDB servers is zero. AMEN to this one. Very nice answer. It's the way we run infra.

dlidstrom 6y ago
What does this mean?

tomerbd 6y ago

I wish I knew to do all this, is there any guide best practice that I can read like a book that includes all above?

hardwaresofton 6y ago

So if you'll indulge me -- this list is exactly why a system like Kubernetes is valuable and why I think personally that it contains a lot of essential complexity. Kubernetes attempts to do all of the above, which is why it's so massive, and I'd argue it's actually less complex than knowing all the tools above -- but it's an equal degree less universally applicable. In this way, it's perfect for the dev who never wants to "do ops", and less so for the dev that already knows ops (or any regular sysadmin/ops person), because they already know all these answers.

15155 6y ago
Devops person here: I already know all of these answers and would still choose Kubernetes over hand-rolling all of this again and again. "Just use Packer, some AMIs, some ASGs, some CF templates, some ELBs, some EC2 instances!" No thanks: I'll Terraform an EKS cluster in 30 lines of HCL and deploy my applications with a Dockerfile and a handful of YAML files.
- hardwaresofton 6y ago
  > CF Templates Yeah, this is where it kicked in for me. Never mind the fact that all of that is AWS specific and absolutely doesn't help you if you ever move clouds. Great to know all the stuff below, but Kubernetes is a wonderful abstraction layer above that stuff, and it gets better every day. CF could have become Kubernetes -- it was supposed to be, but it just never got the mixture right (and of course is AWS exclusive).
  - jen20 6y ago
    I find the ridiculous false dichotomy between Terraform for Kubernetes and Cloudformation for more basic infrastructure even more ironic given that I am still the eighth most prolific contributor to Terraform _over three years after leaving HashiCorp_.
- ex_amazon_sde 6y ago
  Both options are questionable.

lazzlazzlazz 6y ago

Thanks for this great list of answers. I was startled to see someone vomiting a list of unresearched questions as if they constituted a rebuttal. "Taking the bait" was the right call. Thanks again.

karakanb 6y ago

Would you mind pointing out some examples that you can achieve these on a bare-bones installation in an automated manner? Like, I mean sharing some real examples that I can install and forget. You can deploy the simplest "hello world" webserver in any language of your choice. >> How do you run an application on a cluster of plain old linux machines? > Build a package, install it in an image, run that image in an autoscaling group (or whatever equivalent your cloud of choice offers). How come it is any different than running the Docker image on Kubernetes? The image build process is the same, delegating running the image to the platform is the same, the only difference at this point is the name of the platform you are running. Even if you were deploying ZIP archives to Elastic Beanstalk, if it doesn't work as expected, you'd have to debug it as an outsider, and you'd still have to know about the technology. I don't see how it is any different than Kubernetes. >> How do you update your app without downtime? > Blue-green deployment, or phased rollout. How exactly? There are gazillion ways of doing them, they are rough concepts, what we need is a reliably working setup that requires as much effort from us as possible, and there are absolutely no standards on how to do them. Are you going to use Ansible? Maybe just SSH into the node and change the symlink? Maybe some other ways? >> How do you replicate your environment if you want to add a new server to your cluster? > Start the server from the same image. How do you do that? You'd either do that manually on AWS console, or build some tooling to achieve that. If you were to do that via the autoscaling options the vendor is providing, then it is no different than Kubernetes: if that doesn't work then you'd have to debug regardless of the platform that is managing the autoscaling. >> If your app has microservices how do services discover each other? > Consul, or DNS, depending on your appetite What is the difference between trying to learn how does Consul handle service discovery vs how does Kubernetes handle it? >> How do you mount volumes from cloud storage? > It's a bit unclear exactly what you mean here, but I'll assume you mean either block devices (just attach them at machine boot, or on startup if they need a claim), or NFS. Would you mind sharing examples that are not vendor-specific and that'd be configurable on a per-service fashion easily, hopefully without writing any code? >> How do you update configuration? > Either update Consul and have it propagate configuration, or update a configuration package and push it out. How is this any better than pushing your changes to Kubernetes? I personally don't know how does Consul work or how to update a configuration package and push it out to somewhere, I don't even know where to push them. In this context, learning them is also not any better than learning how to do them on Kubernetes. >> How do you automatically restart failed applications? > Systemd restart policy. So, this means that you'd need to learn how to utilize Systemd properly in order to be able to start running your application and write the configuration for that somewhere, and also deal with propagating that configuration to all the machines. >> How do you monitor if your applications are working? > From outside - something like pingdom, and some kind of continuous testing. It's critical that this is measured from the perspective of a user. The question was not really that. The tools like pingdom won't help you if an internal dependency of your application starts failing suddenly. You need a standardized solution for gathering various standard metrics from various services of yours, things like request rate, error rate, request durations, as well as defining custom metrics on the application level such as open database connections, latencies on dependencies, and so on. You will definitely need a proper metrics solution for running any serious workload, and in addition to that you'll also want to be able to alert on some of these metrics. There is no standardized solution for these problems, which means you'll need to roll your own. >> How do you view your log files remotely? > Cloudwatch, Syslog, SSH (depending on requirements). The proper alternative to the Kubernetes' solution is Cloudwatch, and even then the simplicity of `kubectl logs <pod_name>` is still better than trying to understand how Cloudwatch works. >> How do you port-forward from localhost to your Linux server to test your app locally? > SSH. This is not a trivial setup. Let's say you have a service A running remotely but it is not exposed, meaning that you cannot reach it from your local machine, and you'd like to be able to use that while developing your service B locally, how would you set this up in an easy way? The points regarding the images are the same points as any Docker image, so it really boils down to the choice and one doesn't have an advantage over the other in this context. What I am trying to say is: there are quite a lot of problems when running any kind of serious workload, and there are thousands of alternative combinations for solving them, and they were solved even before Kubernetes existed; however, there were no standardized way of doing things, and that's what Kubernetes is allowing people to do. There are definitely downsides of Kubernetes, but trying to point specific examples like these don't help as they are just names of individual software that also have learning curve and they all operate differently. I do wish there was a simpler solution, I wish Docker Swarm succeeded as a strong alternative for Kubernetes for simpler cases as it is brilliant working locally, and I wish we didn't have to deal with all these problems, but it is what it is. As of today, I can write a Golang web application, prepare a 10-lines Dockerfile, write ~50 lines of YAML and I am good to go: I can deploy this application on any Kubernetes cluster on any cloud provider and have all these stuff defined above automatically. Do I need to add a Python application alongside: I just write another 20-lines Dockerfile for that application, again ~50 lines of YAML for Kubernetes deployment and bam, that's it. For both of these services I have automated recovery, load balancing, auto-scaling, rolling deployments, stateless deployments, aggregated logging, without writing any code for any tooling.

Longwelwind 6y ago

I feel like your post describes exactly what Kubernetes and container images would bring to your infra. If you were to deploy a solution like you described, you would get something more complex than simply running Kubernetes, except worse. I suspect that you believe your solution would be simpler only because you are more comfortable with those technologies than with k8s. The more I read criticism of k8s, the more I'm persuaded that what people calls "old boring technologies" truly is "technologies I'm comfortable with". On top of that, you'd need to separately document everything you do on your infra. The advantage of Docker images over AMI is that you have a file that describes how the image was built. With an AMI, you would need to hope that the guy who created the AMI documented it somewhere (or hope that he has not quit). Same goes for k8s, where configurations are committed into your repository. At the end of the day, k8s stays a tool that you should use only when needed (and also if you have the capabilities of using it), but I think you shouldn't discard it simply because you are capable of producing the same result by other means. You get a lot more by using k8s, in my opinion.

jen20 6y ago
> The advantage of Docker images over AMI is that you have a file that describes how the image was built. https://packer.io https://packer.io - how is this even a discussion? > the more I'm persuaded that what people calls "old boring technologies" truly is "technologies I'm comfortable with". My infrastructure runs in Nomad on bare metal. I am by no means opposed to “progress”, I just don’t think Kubernetes is the be-all-and-end-all of infrastructure and would like to have a less hysterical debate about it than the parent to my original post presented.

lenkite 6y ago

I loved this list of succinct answers.

sciurus 6y ago

Great answers. To be fair though, that's not "Plain old linux" like zaptheimpaler suggested was somehow possible. That's linux plus AWS managed services plus software from Hashicorp. Which is a great stack to be on, but has its own complexities and tradeoffs.