5 ms·

> Plain old linux is the alternative How do you run an application on a cluster of plain old linux machines? How do you do load balancing? How do you scale up

> Plain old linux is the alternative How do you run an application on a cluster of plain old linux machines? How do you do load balancing? How do you scale up and down? How do you update your app without downtime? How do you roll back easily if something goes wrong? How do you ensure all your servers are running the same version of dependencies? How do you update those dependencies? How do you replicate your environment if you want to add a new server to your cluster? If your app has microservices how do services discover each other? How do you mount volumes from cloud storage? How do you update configuration? How do you automatically restart failed applications? How do you monitor if your applications are working? How do you make sure the right number of MongoDB replicas are running at all times? How do you view your log files remotely? How do you port-forward from localhost to your Linux server to test your app locally?

verroq 6y ago
The easy answer is that you don’t need to run your service on distributed across 1000 VPS instances. A handful of dedicated machines is enough. For example: stackoverflow. None of these are difficult to answer. You can set up automated deployment/rollback however you want. You don’t care about the dependencies your code is compiled and linux has binary compat. You don’t need to split your app into 1000 unmanageable interdependent microservices. You have enough disk space on your dedicated machine or use NFS. You set up your systemd service file to auto restart your service. Etc etc. When you have k8 as a hammer everything looks like a nail.
ditonal 6y ago
The difficulty of all that stuff on plain old Linux is overstated and the difficulty of doing that on K8s is understated. And I agree with grandfathers point that if you don’t understand how to do that on plain old Linux you will struggle on K8s. K8s is ok for huge enterprises with tons of engineers but somehow K8s advocates make it seem like learning and operating nginx on ubuntu is this huge challenge when it’s usually not.
- rumanator 6y ago
  Your comment makes no sense at all. Kubernetes is not equivalent to running nginx on Ubuntu. This comparison is simply too absurd to the point I really doubt you have any idea of what Kubernetes is designed to do and why people use it. I mean, who in their right mind would even describe a cluster of VMs and/or COTS hardware running with a VPN and designed to automatically run and manage the lifecycle of containers as well as the computational resources they consume as... Operating nginx on Ubuntu? If you don't have anything to add to the discussion and your only relationship with the topic is that you are completely unfamiliar and are intimidated by what you don't know then please refrain from increasing the signal-to-noise ratio of the discussion.
  - anang 6y ago
    I think the point is more that the complexity of a cluster of VMs that manage the lifecycle of containers is often overkill for a service that would work with nginx installed on Ubuntu, and that often times the former is sold as reducing complexity and the latter as increasing it.
  - redis_mlc 6y ago
    rumanator: the mistake is yours. Please read the parent post again, since you missed any nuance. I agree with said parent. You can go a long way with ASGs, iptables and yum install - literally the entire first decade of your startup. In larger companies, just the politics introduced with managing k8s by another dept. is mind-numbing.
    - rumanator 6y ago
      No, the assertion that nginx on ubuntu is equivalent to kubernetes os mind-numbingly wrong, for starters for being entirely and completely oblivious to containers. The comparison isn't even wrong: it simply makes no sense at all. And no, being able to run software is not equivalent to Kubernetes. It's not even in the same ballpark of the discussion. You are entirely free to go full Amish on your pet projects but let's not pretend that managing a pet server brings any operational advantage over, say, just being able to seamlessly deploy N versions of your app with out-of-the-box support for blue/green deployments and with the help of a fully versioned system that allows to undo and resume operating with a past configuration. You don't do that by flaunting iptables and yum, do you? If anyone needs to operate a cluster then they need tooling to manage and operate a cluster. Your iptables script isn't it.
yjftsjthsd-h 6y ago
DNS round-robin sends inbound to 2 haproxy servers, which proxy traffic on to the rest of the cluster. Scaling means "add another box, install your app, add haproxy entry". Service discovery is just an internal-facing haproxy front. If you must have "volumes from cloud storage", you use NFS (but if you can help it, you don't do that). Updates, restarts (including zero-downtime courtesy of draining the node out of haproxy), etc. are all quite doable with SSH and shell scripts. You run an actual monitoring system, because it's not like k8s could monitor server hardware anyways. Likewise, syslog is not exactly novel. I... don't understand why you're port forwarding? Either run on the dev cluster or your laptop. So yes, you'd need a handful of different systems, but "k8s" is hardly a single system once you have to actually manage it, and most of the parts are simpler.
- p_l 6y ago
  Having done exactly that in the past, both by hand and with configuration management (including custom scripting that synchronized HAproxy configs etc.), I would say that I can do all of that much, much simpler in k8s. Installing application and routing it with load balancing, TLS etc and support for draining is really simplified, something that used to require annoying amount of time (it got better as Lua scripting was extended in HAproxy, but by that time I also had k8s). Resource allocation, including persistent storage (including NFS) is so much easier it's not funny, it makes my old years painful to think about. Syslog is not novel but getting all applications to log the same way was always a PITA, and at least with containers it's a bit easier (I still have to sometimes ship physical logs from containers...). As for monitoring, that's one of the newer and more advanced topics, but it's possible to integrate more complex server monitoring with k8s - it already provides a lot of OS/application side monitoring that really makes it easier to setup observability - but now there's quite simple way to integrate custom node states into kubernetes allowing a quick glance way to check why a node is not accepting jobs, integrated with the system so you can actually trigger from a health-check that the node is in trouble and should not take jobs.
deleted 6y ago
[deleted]
rapsey 6y ago
Tangentially lots of those features are quite nicely handled by Erlang. Using plain old Linux and Erlang/Elixir as a backend you can get extremely far with scaling with very little overhead and issues. It is unfortunate that so many ignore the benefits of using Erlang as a backend technology.
lazzlazzlazz 6y ago
This is a toxic trolling technique called "sealioning". Thankfully, `jen20` was kind enough to take your questions in good faith and made a good demonstration of this confusion.
jen20 6y ago
These are commonly raised concerns, all of which have answers much simpler than "install this giant distributed system". I'll go ahead and answer them since I take the questions to be in good faith... > How do you run an application on a cluster of plain old linux machines? Build a package, install it in an image, run that image in an autoscaling group (or whatever equivalent your cloud of choice offers). > How do you do load balancing? An Elastic Load Balancer (v1 or v2), HAProxy, an F5 - this is deployment environment specific (just like in Kubernetes). > How do you update your app without downtime? Blue-green deployment, or phased rollout. > How do you ensure all your servers are running the same version of dependencies? Build them from a common image. > How do you update those dependencies? Update the Packer template that builds that image. > How do you replicate your environment if you want to add a new server to your cluster? Start the server from the same image. > If your app has microservices how do services discover each other? Consul, or DNS, depending on your appetite. > How do you mount volumes from cloud storage? It's a bit unclear exactly what you mean here, but I'll assume you mean either block devices (just attach them at machine boot, or on startup if they need a claim), or NFS. > How do you update configuration? Either update Consul and have it propagate configuration, or update a configuration package and push it out. > How do you automatically restart failed applications? Systemd restart policy. > How do you monitor if your applications are working? From outside - something like pingdom, and some kind of continuous testing. It's critical that this is measured from the perspective of a user. > How do you make sure the right number of MongoDB replicas are running at all times? Somewhat flippant answer here: the right number of MongoDB servers is zero. More generally, by limiting the size of an autoscaling group. > How do you view your log files remotely? Cloudwatch, Syslog, SSH (depending on requirements). > How do you port-forward from localhost to your Linux server to test your app locally? SSH.
- trepatudo 6y ago
  Thank you. I use most of this, I've been using it for years and I just don't talk about it because it's hard to argue when people just want to force an idea that k8s is "really the best way of doing things". Also, haproxy is one of the most reliable software I've ever used.
  - 6y ago
OOPMan 6y ago
I call bullshit on any 1-man team needing to worry much about this stuff. Also, see jen20's response. If your only tool is a hammer (called k8s) then I'm sure everything really does look like a nail...
ex_amazon_sde 6y ago
This is how Amazon works internally, and by choice, because unnecessary complexity is the enemy. > How do you do load balancing? With dedicated load balancers. They are even cheaper than servers because they are optimized for doing one thing. > How do you scale up and down? With pretty trivial automation than trigger deployments when load is too high. > How do you update your app without downtime? (app?!) A package manager pulls updated packages, deploys them and restarts the services. The LB does the flipping. > How do you roll back easily if something goes wrong? The package manager does that quite easily > How do you ensure all your servers are running the same version of dependencies? > How do you update those dependencies? Package managers support versioning on dependencies. > How do you replicate your environment if you want to add a new server to your cluster? You install MyStuff version 1.2.3 and everything goes in. > If your app has microservices how do services discover each other? Thankfully Amazon uses services that are not micro. DNS does the trick. > How do you update configuration? It's in the packages. > How do you automatically restart failed applications? OSes do that since decades. > How do you monitor if your applications are working? LBs check the endpoints, monitoring tools do the rest. Essentially you can build all of this with the technology that existed 15 years ago. It works reliably and it takes much less learning.

verroq 6y ago

The easy answer is that you don’t need to run your service on distributed across 1000 VPS instances. A handful of dedicated machines is enough. For example: stackoverflow. None of these are difficult to answer. You can set up automated deployment/rollback however you want. You don’t care about the dependencies your code is compiled and linux has binary compat. You don’t need to split your app into 1000 unmanageable interdependent microservices. You have enough disk space on your dedicated machine or use NFS. You set up your systemd service file to auto restart your service. Etc etc. When you have k8 as a hammer everything looks like a nail.

ditonal 6y ago

The difficulty of all that stuff on plain old Linux is overstated and the difficulty of doing that on K8s is understated. And I agree with grandfathers point that if you don’t understand how to do that on plain old Linux you will struggle on K8s. K8s is ok for huge enterprises with tons of engineers but somehow K8s advocates make it seem like learning and operating nginx on ubuntu is this huge challenge when it’s usually not.

rumanator 6y ago
Your comment makes no sense at all. Kubernetes is not equivalent to running nginx on Ubuntu. This comparison is simply too absurd to the point I really doubt you have any idea of what Kubernetes is designed to do and why people use it. I mean, who in their right mind would even describe a cluster of VMs and/or COTS hardware running with a VPN and designed to automatically run and manage the lifecycle of containers as well as the computational resources they consume as... Operating nginx on Ubuntu? If you don't have anything to add to the discussion and your only relationship with the topic is that you are completely unfamiliar and are intimidated by what you don't know then please refrain from increasing the signal-to-noise ratio of the discussion.
- anang 6y ago
  I think the point is more that the complexity of a cluster of VMs that manage the lifecycle of containers is often overkill for a service that would work with nginx installed on Ubuntu, and that often times the former is sold as reducing complexity and the latter as increasing it.
- redis_mlc 6y ago
  rumanator: the mistake is yours. Please read the parent post again, since you missed any nuance. I agree with said parent. You can go a long way with ASGs, iptables and yum install - literally the entire first decade of your startup. In larger companies, just the politics introduced with managing k8s by another dept. is mind-numbing.
  - rumanator 6y ago
    No, the assertion that nginx on ubuntu is equivalent to kubernetes os mind-numbingly wrong, for starters for being entirely and completely oblivious to containers. The comparison isn't even wrong: it simply makes no sense at all. And no, being able to run software is not equivalent to Kubernetes. It's not even in the same ballpark of the discussion. You are entirely free to go full Amish on your pet projects but let's not pretend that managing a pet server brings any operational advantage over, say, just being able to seamlessly deploy N versions of your app with out-of-the-box support for blue/green deployments and with the help of a fully versioned system that allows to undo and resume operating with a past configuration. You don't do that by flaunting iptables and yum, do you? If anyone needs to operate a cluster then they need tooling to manage and operate a cluster. Your iptables script isn't it.

yjftsjthsd-h 6y ago

DNS round-robin sends inbound to 2 haproxy servers, which proxy traffic on to the rest of the cluster. Scaling means "add another box, install your app, add haproxy entry". Service discovery is just an internal-facing haproxy front. If you must have "volumes from cloud storage", you use NFS (but if you can help it, you don't do that). Updates, restarts (including zero-downtime courtesy of draining the node out of haproxy), etc. are all quite doable with SSH and shell scripts. You run an actual monitoring system, because it's not like k8s could monitor server hardware anyways. Likewise, syslog is not exactly novel. I... don't understand why you're port forwarding? Either run on the dev cluster or your laptop. So yes, you'd need a handful of different systems, but "k8s" is hardly a single system once you have to actually manage it, and most of the parts are simpler.

p_l 6y ago
Having done exactly that in the past, both by hand and with configuration management (including custom scripting that synchronized HAproxy configs etc.), I would say that I can do all of that much, much simpler in k8s. Installing application and routing it with load balancing, TLS etc and support for draining is really simplified, something that used to require annoying amount of time (it got better as Lua scripting was extended in HAproxy, but by that time I also had k8s). Resource allocation, including persistent storage (including NFS) is so much easier it's not funny, it makes my old years painful to think about. Syslog is not novel but getting all applications to log the same way was always a PITA, and at least with containers it's a bit easier (I still have to sometimes ship physical logs from containers...). As for monitoring, that's one of the newer and more advanced topics, but it's possible to integrate more complex server monitoring with k8s - it already provides a lot of OS/application side monitoring that really makes it easier to setup observability - but now there's quite simple way to integrate custom node states into kubernetes allowing a quick glance way to check why a node is not accepting jobs, integrated with the system so you can actually trigger from a health-check that the node is in trouble and should not take jobs.

deleted 6y ago

[deleted]

rapsey 6y ago

Tangentially lots of those features are quite nicely handled by Erlang. Using plain old Linux and Erlang/Elixir as a backend you can get extremely far with scaling with very little overhead and issues. It is unfortunate that so many ignore the benefits of using Erlang as a backend technology.

lazzlazzlazz 6y ago

This is a toxic trolling technique called "sealioning". Thankfully, `jen20` was kind enough to take your questions in good faith and made a good demonstration of this confusion.

jen20 6y ago

These are commonly raised concerns, all of which have answers much simpler than "install this giant distributed system". I'll go ahead and answer them since I take the questions to be in good faith... > How do you run an application on a cluster of plain old linux machines? Build a package, install it in an image, run that image in an autoscaling group (or whatever equivalent your cloud of choice offers). > How do you do load balancing? An Elastic Load Balancer (v1 or v2), HAProxy, an F5 - this is deployment environment specific (just like in Kubernetes). > How do you update your app without downtime? Blue-green deployment, or phased rollout. > How do you ensure all your servers are running the same version of dependencies? Build them from a common image. > How do you update those dependencies? Update the Packer template that builds that image. > How do you replicate your environment if you want to add a new server to your cluster? Start the server from the same image. > If your app has microservices how do services discover each other? Consul, or DNS, depending on your appetite. > How do you mount volumes from cloud storage? It's a bit unclear exactly what you mean here, but I'll assume you mean either block devices (just attach them at machine boot, or on startup if they need a claim), or NFS. > How do you update configuration? Either update Consul and have it propagate configuration, or update a configuration package and push it out. > How do you automatically restart failed applications? Systemd restart policy. > How do you monitor if your applications are working? From outside - something like pingdom, and some kind of continuous testing. It's critical that this is measured from the perspective of a user. > How do you make sure the right number of MongoDB replicas are running at all times? Somewhat flippant answer here: the right number of MongoDB servers is zero. More generally, by limiting the size of an autoscaling group. > How do you view your log files remotely? Cloudwatch, Syslog, SSH (depending on requirements). > How do you port-forward from localhost to your Linux server to test your app locally? SSH.

trepatudo 6y ago
Thank you. I use most of this, I've been using it for years and I just don't talk about it because it's hard to argue when people just want to force an idea that k8s is "really the best way of doing things". Also, haproxy is one of the most reliable software I've ever used.
- 6y ago

OOPMan 6y ago

I call bullshit on any 1-man team needing to worry much about this stuff. Also, see jen20's response. If your only tool is a hammer (called k8s) then I'm sure everything really does look like a nail...

ex_amazon_sde 6y ago

This is how Amazon works internally, and by choice, because unnecessary complexity is the enemy. > How do you do load balancing? With dedicated load balancers. They are even cheaper than servers because they are optimized for doing one thing. > How do you scale up and down? With pretty trivial automation than trigger deployments when load is too high. > How do you update your app without downtime? (app?!) A package manager pulls updated packages, deploys them and restarts the services. The LB does the flipping. > How do you roll back easily if something goes wrong? The package manager does that quite easily > How do you ensure all your servers are running the same version of dependencies? > How do you update those dependencies? Package managers support versioning on dependencies. > How do you replicate your environment if you want to add a new server to your cluster? You install MyStuff version 1.2.3 and everything goes in. > If your app has microservices how do services discover each other? Thankfully Amazon uses services that are not micro. DNS does the trick. > How do you update configuration? It's in the packages. > How do you automatically restart failed applications? OSes do that since decades. > How do you monitor if your applications are working? LBs check the endpoints, monitoring tools do the rest. Essentially you can build all of this with the technology that existed 15 years ago. It works reliably and it takes much less learning.