4 ms·

My 2c: you don't need K8s unless you are google-scale. Even if you think you are google-scale, you are not. Maintenance-wise and $$-wise, two VPS boxes with Clo

by dig1 4y ago

My 2c: you don't need K8s unless you are google-scale. Even if you think you are google-scale, you are not. Maintenance-wise and $$-wise, two VPS boxes with Cloudflare (even with an enterprise account) setup is usually cheaper than an ordinary K8s setup. But again, it all depends on your use case, and people usually overestimate their use cases and the company's growth.

KronisLV 4y ago
> Maintenance-wise and $$-wise, two VPS boxes with Cloudflare (even with an enterprise account) setup is usually cheaper than an ordinary K8s setup. How do you feel about the benefits of using containers? And if you do use containers, how do you feel about the benefits of orchestrating them, instead of running something like Docker Compose? E.g. having overlay networks and being able to deploy a new version of your software on multiple nodes simultaneously? I've found that even when you don't want K8s, something like Docker Swarm or Hashicorp Nomad may still have benefits for you, depending on what you're trying to do. Swarm is basically feature-complete, boring and just like adding multi-node capabilities on top of Docker Compose anyways. Nothing to install apart from Docker and nothing to configure, apart from executing a cluster init/join command.
- dig1 4y ago
  My tool of choice is usually Ansible, so, from the Ansible point of view, deploying to a container, VPS, bare server, or a cluster, looks pretty much the same. I haven't mentioned Swarm intentionally, which IMHO I consider "the next iteration" of your containerized setup: Swarm is a boring, just-works, tool if you need to spread multiple containers, add some networking between them, and have to manage everything by a normal person. And boring is good :)
  - KronisLV 4y ago
    > My tool of choice is usually Ansible, so, from the Ansible point of view, deploying to a container, VPS, bare server, or a cluster, looks pretty much the same. I agree that Ansible is an excellent tool! Though personally, I enjoy establishing a clear boundary between the "infrastructure" and "app" parts of any setup - the former ensuring that OS has whatever runtimes are necessary for any given application to run, user accounts, groups, folders, services etc., whereas the latter is whatever apps are running on the server. So I'd use Ansible for most of the former, setup a container cluster and then use something like https://docs.ansible.com/ansible/latest/collections/community/docker/docker_swarm_module.html#ansible-collections-community-docker-docker-swarm-module https://docs.ansible.com/ansible/latest/collections/communit... to manage actual application deployments. Why? To make the apps more throwaway and limit the fallout in case of bad configurations/deployments, as well as make sharing the base Ansible playbooks for what constitutes a production ready server setup easier amongst projects. Of course, you could as well use Ansible to install something like JDK/Tomcat and set up its configuration which worked well for me in the past, but personally that didn't scale quite as well as running Java/Node/Ruby/Python/PHP in containers and approaching them with Ansible almost like black boxes (e.g. copy a bunch of files, like secrets, into these directories, deploy some arbitrary YAML against the Docker Swarm cluster), which was surprisingly easy to do.
- slotrans 4y ago
  > And if you do use containers, how do you feel about the benefits of orchestrating them Why would I need that? A container is just a fancy executable. No one ever talked about "executable orchestration". Container orchestration does not need to be a thing. If you need anything more complex than `docker run` you are either extremely large and this whole discussion is irrelevant, or you need to question your life choices.
  - KronisLV 4y ago
    > A container is just a fancy executable. Not just that, but containers also have a lot of knowledge and tooling around them for running services more painlessly. I suggest that you familiarize yourself with this excellent site: https://12factor.net/ https://12factor.net/ > No one ever talked about "executable orchestration". Actually thousands of collective developer-years have gone into developing Java EE, OSGi and also figuring out how to run "containers" inside of Tomcat, GlassFish and many other web servers, which basically were executables or modules that contained application logic that had to be run. Both in the Java ecosystem and many others, it was eventually decided that attempts like this (as well as going in the separate direction and shipping VMs or VM images, a la Vagrant) don't really work that nicely and containers were a middle ground that people settled on - unified bundles of almost everything (sans kernel) that any given application might need to run. Of course, other interesting projects like Flatpak, AppImage, snaps and even functions-as-a-service all concern themselves with how any piece of executable code should be run, especially in the case of latter. And then there are systems which attempt to distribute work across any number of worker nodes, actually you can just look at the entire HPC industry. Thus, I believe that one can definitely say that a plethora of options for running things has been explored and will continue to be something that people research and iterate upon in the future! > Container orchestration does not need to be a thing. That's a lot like saying that OpenRC, systemd or other init systems shouldn't be a thing. Of course people are going to want to explore standardized options for figuring out how to organize their software that should be running! Even more so when you're running it across multiple nodes and the risks posed by human error are great! Just look at what happened to Knight Capital because of human error: https://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/ https://dougseven.com/2014/04/17/knightmare-a-devops-caution... > If you need anything more complex than `docker run` you are either extremely large and this whole discussion is irrelevant, or you need to question your life choices. I believe that this is an unnecessarily dismissive tone and a misrepresentation of what scales of work might benefit from container orchestration. Not all of it needs to be as complex as Kubernetes. Frankly, in many cases, the likes of Docker Swarm will suffice, much like you might also run Docker Compose locally so you don't have to muck about with 5-10 separate Docker run commands just to launch all of the dependencies of an app that you're developing, or software that someone else has written and that you'd like to use. But if you're running all of your software in prod on a single node, don't have any horizontally scaled services, or prefer to do everything manually, such as rolling out updates on a per-service basis, then the benefits that you'll get from containers will indeed mostly regard their runtimes, configuration, logging, resource limits and similar qualities, rather than any degree of automation, because you'll miss out on that.
thiht 4y ago
I see your point but you might need Kubernetes before going « Google scale ». Google is probably the biggest scale in the world, that would mean they’re the only one requiring Kubernetes, which they’re not. A big advantage of Kubernetes is that it can help to harmonize the deployment process of numerous teams in a whole company. As long as your company has multiple teams, including a team dedicated to administrating the Kubernetes cluster(s), then it can be a good choice. Because at this point, the cheapest solution is not necessarily cheap anymore. That’s NOT a problem in startups, even less in early stage startups though.
manigandham 4y ago
Kubernetes is not for Google scale. At that size you have completely custom infrastructure which is why Google continues to use Borg (and it's evolutions) instead. K8S is specifically designed for the non-Google's to run with similar capabilities and features. Most anti-K8S sentiment comes from the overhead of installing and running it (vs using a managed service) and those that just never learned it. By the time you setup the same functionality, you will end up spending more work on a less reliable and agile setup.
- icod 4y ago
  "they" never learn because it's too complex to setup and maintain. A managed service is too expensive, up to factor 200 prices for outgoing traffic vs a dedicated server with flat traffic (e.g. Hetzner)
mmcnl 4y ago
K8s is not about scaling, it's about declarative configuration management, which gives you the opportunity to easily scale as an added bonus. It feels extremely liberating to have my code and Kubernetes files in a repository, knowing that I can deploy this with one command to production on any Kubernetes cluster. It's a great infrastructure abstraction layer. I agree it's expensive though. A docker-compose.yml on a VPS usually works fine as well. Requires some additional configuration outside the Docker ecosystem though.
- deleted 4y ago
  [deleted]
- icod 4y ago
  what's wrong with git clone go build systemctl restart my-service Why would I want to proxy through docker?
  - mmcnl 4y ago
    This question refers to the benefits of containerization and not Kubernetes. But to answer your question: your command is dependent on (1) the availability of Git, (2) Go and (3) systemd on your host OS. Without containers, this list of dependencies is dynamic and can change in size for every project. With containers, the list of dependencies reduces to one dependency only for every project: Docker.

by dig1 4y ago

KronisLV 4y ago

> Maintenance-wise and $$-wise, two VPS boxes with Cloudflare (even with an enterprise account) setup is usually cheaper than an ordinary K8s setup. How do you feel about the benefits of using containers? And if you do use containers, how do you feel about the benefits of orchestrating them, instead of running something like Docker Compose? E.g. having overlay networks and being able to deploy a new version of your software on multiple nodes simultaneously? I've found that even when you don't want K8s, something like Docker Swarm or Hashicorp Nomad may still have benefits for you, depending on what you're trying to do. Swarm is basically feature-complete, boring and just like adding multi-node capabilities on top of Docker Compose anyways. Nothing to install apart from Docker and nothing to configure, apart from executing a cluster init/join command.

dig1 4y ago
My tool of choice is usually Ansible, so, from the Ansible point of view, deploying to a container, VPS, bare server, or a cluster, looks pretty much the same. I haven't mentioned Swarm intentionally, which IMHO I consider "the next iteration" of your containerized setup: Swarm is a boring, just-works, tool if you need to spread multiple containers, add some networking between them, and have to manage everything by a normal person. And boring is good :)
- KronisLV 4y ago
  > My tool of choice is usually Ansible, so, from the Ansible point of view, deploying to a container, VPS, bare server, or a cluster, looks pretty much the same. I agree that Ansible is an excellent tool! Though personally, I enjoy establishing a clear boundary between the "infrastructure" and "app" parts of any setup - the former ensuring that OS has whatever runtimes are necessary for any given application to run, user accounts, groups, folders, services etc., whereas the latter is whatever apps are running on the server. So I'd use Ansible for most of the former, setup a container cluster and then use something like https://docs.ansible.com/ansible/latest/collections/community/docker/docker_swarm_module.html#ansible-collections-community-docker-docker-swarm-module https://docs.ansible.com/ansible/latest/collections/communit... to manage actual application deployments. Why? To make the apps more throwaway and limit the fallout in case of bad configurations/deployments, as well as make sharing the base Ansible playbooks for what constitutes a production ready server setup easier amongst projects. Of course, you could as well use Ansible to install something like JDK/Tomcat and set up its configuration which worked well for me in the past, but personally that didn't scale quite as well as running Java/Node/Ruby/Python/PHP in containers and approaching them with Ansible almost like black boxes (e.g. copy a bunch of files, like secrets, into these directories, deploy some arbitrary YAML against the Docker Swarm cluster), which was surprisingly easy to do.
slotrans 4y ago
> And if you do use containers, how do you feel about the benefits of orchestrating them Why would I need that? A container is just a fancy executable. No one ever talked about "executable orchestration". Container orchestration does not need to be a thing. If you need anything more complex than `docker run` you are either extremely large and this whole discussion is irrelevant, or you need to question your life choices.
- KronisLV 4y ago
  > A container is just a fancy executable. Not just that, but containers also have a lot of knowledge and tooling around them for running services more painlessly. I suggest that you familiarize yourself with this excellent site: https://12factor.net/ https://12factor.net/ > No one ever talked about "executable orchestration". Actually thousands of collective developer-years have gone into developing Java EE, OSGi and also figuring out how to run "containers" inside of Tomcat, GlassFish and many other web servers, which basically were executables or modules that contained application logic that had to be run. Both in the Java ecosystem and many others, it was eventually decided that attempts like this (as well as going in the separate direction and shipping VMs or VM images, a la Vagrant) don't really work that nicely and containers were a middle ground that people settled on - unified bundles of almost everything (sans kernel) that any given application might need to run. Of course, other interesting projects like Flatpak, AppImage, snaps and even functions-as-a-service all concern themselves with how any piece of executable code should be run, especially in the case of latter. And then there are systems which attempt to distribute work across any number of worker nodes, actually you can just look at the entire HPC industry. Thus, I believe that one can definitely say that a plethora of options for running things has been explored and will continue to be something that people research and iterate upon in the future! > Container orchestration does not need to be a thing. That's a lot like saying that OpenRC, systemd or other init systems shouldn't be a thing. Of course people are going to want to explore standardized options for figuring out how to organize their software that should be running! Even more so when you're running it across multiple nodes and the risks posed by human error are great! Just look at what happened to Knight Capital because of human error: https://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/ https://dougseven.com/2014/04/17/knightmare-a-devops-caution... > If you need anything more complex than `docker run` you are either extremely large and this whole discussion is irrelevant, or you need to question your life choices. I believe that this is an unnecessarily dismissive tone and a misrepresentation of what scales of work might benefit from container orchestration. Not all of it needs to be as complex as Kubernetes. Frankly, in many cases, the likes of Docker Swarm will suffice, much like you might also run Docker Compose locally so you don't have to muck about with 5-10 separate Docker run commands just to launch all of the dependencies of an app that you're developing, or software that someone else has written and that you'd like to use. But if you're running all of your software in prod on a single node, don't have any horizontally scaled services, or prefer to do everything manually, such as rolling out updates on a per-service basis, then the benefits that you'll get from containers will indeed mostly regard their runtimes, configuration, logging, resource limits and similar qualities, rather than any degree of automation, because you'll miss out on that.

thiht 4y ago

I see your point but you might need Kubernetes before going « Google scale ». Google is probably the biggest scale in the world, that would mean they’re the only one requiring Kubernetes, which they’re not. A big advantage of Kubernetes is that it can help to harmonize the deployment process of numerous teams in a whole company. As long as your company has multiple teams, including a team dedicated to administrating the Kubernetes cluster(s), then it can be a good choice. Because at this point, the cheapest solution is not necessarily cheap anymore. That’s NOT a problem in startups, even less in early stage startups though.

manigandham 4y ago

Kubernetes is not for Google scale. At that size you have completely custom infrastructure which is why Google continues to use Borg (and it's evolutions) instead. K8S is specifically designed for the non-Google's to run with similar capabilities and features. Most anti-K8S sentiment comes from the overhead of installing and running it (vs using a managed service) and those that just never learned it. By the time you setup the same functionality, you will end up spending more work on a less reliable and agile setup.

icod 4y ago
"they" never learn because it's too complex to setup and maintain. A managed service is too expensive, up to factor 200 prices for outgoing traffic vs a dedicated server with flat traffic (e.g. Hetzner)

mmcnl 4y ago

K8s is not about scaling, it's about declarative configuration management, which gives you the opportunity to easily scale as an added bonus. It feels extremely liberating to have my code and Kubernetes files in a repository, knowing that I can deploy this with one command to production on any Kubernetes cluster. It's a great infrastructure abstraction layer. I agree it's expensive though. A docker-compose.yml on a VPS usually works fine as well. Requires some additional configuration outside the Docker ecosystem though.

deleted 4y ago
[deleted]
icod 4y ago
what's wrong with git clone go build systemctl restart my-service Why would I want to proxy through docker?
- mmcnl 4y ago
  This question refers to the benefits of containerization and not Kubernetes. But to answer your question: your command is dependent on (1) the availability of Git, (2) Go and (3) systemd on your host OS. Without containers, this list of dependencies is dynamic and can change in size for every project. With containers, the list of dependencies reduces to one dependency only for every project: Docker.