3 ms·

I did, somewhat. Running between 8 and 14 Linode and Digital ocean VMS, I wanted to research if K8s or any docker setup would be good for me. I have been doin

by berkes 6y ago

I did, somewhat. Running between 8 and 14 Linode and Digital ocean VMS, I wanted to research if K8s or any docker setup would be good for me. I have been doing linux server maintanance for over 16 years now, set-up and grew three webhosting companies. My current VMs are for my own tooling (selfhosted mail, nextcloud, matomo) my startup (mostly servers crunching OSM databases) and some leftover client hosting (mostly Rails and Sinatra). The issues I ran into with Docker and k8s were all small and could probably be overcome given extra time and effort. But combined, I decided that containerization does not solve enough problems for me, that it warrants all the new problems it introduces. In no particular order: * Firewalling: On a normal linux machine or in a cluster a long-solved problem (resp: iptables/ufw or dedicated firewall servers/hardware). With containerized: no idea. Seems a largely unsolved issue, probably because "containerization should not do any firewalling". And partly because of how network mapping works, the problem is mitigated (but not solved!) there. * Monitoring: at least three popular docker images that I used forgot to add logrotation, crashing after long term running. That in itself is bad (and shows lack of detailed finish in many images) but it shows you need to monitor. I used munin and am now migrating towards prometheus/grafana, but I really have no idea how to properly monitor a flock of containers. Another problem that has been solved for ages, but requires time&effort in a containerized env. * Timed jobs (cronjobs). There is tooling to spin-up-and-run containers on timed schedules, but no-one as easy and stable as having Ansible write a file to /etc/cron.d/osm_poi_extract (or use systemd, fine with me). * fail2ban, tripwire, etc: small tooling that is hard to do or requires extra attention in a containerized setup. * unity: if you rely on 3rd party containers you'll quickly have a rainbow-spectrum of machines: Ubuntu LTS, Ubuntu edge, Ubuntu ancient, Alpine, CentOS, a rare freeBSD. The interface is consistent, the underlying tech is not: troubleshooting is a terror if you first have to spend 20 minutes of googling "how do I know what version Alpine I have and how do I get curl on this damned thing to see if elasticsearch is maybe giving results on localhost". I realize that my 16+ years of prior linux-sysadmin experience hinder me here: I'm probably trying to make a containerized setup do the same that I'm used to, but which is not really needed (tripwire, fail2ban, firewalls?). But for me, containers -in production- solve no problem. Yet they introduce a whole range of problems that in a more classic setup have been solved for, sometimes literally, decades. "Infra and state in a documented revision control" is solved mostly with ansible, saltstack, chef or puppet; this can and should certainly evolve and improve. Networking is solved with, well, networking. Hell, one of my previous hosting setups had a cron job that would get "the latest /etc/hosts" hourly: that was our entire DNS setup. it worked. reliably", /etc/hosts and docker, don't get me started (the reply would probably be: but you don't need /etc/hosts in k8s).

jrockway 6y ago
Containers aren't really designed to run without some sort of management infrastructure to handle the problems you mention -- network policy, monitoring, log rotation, scheduling, etc. There are many container runtimes. Kubernetes is popular and solves some of these problems. > Firewalling The base unit in Kubernetes is the Pod, which kind of acts as a mini-VM (without virtualization) that runs 1 or more containers. It's a mini-VM, so has its own network; 127.0.0.1 is the Pod, not the Node that it happens to run on. You can set many networking-related policies at the Pod level; PodSecurityPolicy for very coarse-grained security policies (access to the host network, access to become root, etc.), or NetworkPolicies for what people would traditionally consider firewall rules (only allow access to the MySQL pod from a pod in the MyWebApp service, etc). > Monitoring Kubernetes rotates your logs for you. Retaining logs and making them useful is still an unsolved problem; containerized or not. (I wrote a very complicated log analysis system when I was at Google. I miss it.) Your problems with monitoring (how to get Prometheus to discover containers to scrape) boil down to not having service discovery. Kubernetes provides service discovery, and Prometheus knows how to ask it for a set of endpoints to collect metrics from. (Service discovery is nice in general, and is often something sets of hand-rolled VPCs are missing.) > Timed Jobs Kubernetes has the concept of CronJobs. Overall, I don't like the approach of cron jobs over having a program that is always running and wakes up when it's time to do work. But both options are available to you. > fail2ban, tripwire, etc. Typically people have a broader-scoped access control and rate limiting policy. All external traffic hits a proxy between the outside network and the network that Pods are on. The proxy sends information about the request to an authorization service and receives an access decision, and accepts or rejects the request accordingly. If you had ssh sitting behind such a proxy, you would instruct it to send success/failure information to the authorization service so that it can make an accurate decision for the next connection attempt. There is some desire/work to make ACLs easier and do it at the NetworkPolicy/CNI layer. What you can do largely depends on what CNI plugin your cluster uses, and if you're using a managed k8s provider, you probably don't have much choice in the matter. Hence people replicating the "old world" with authenticating proxies. In general, people are not running SSH servers, especially ones that can accept passwords from the outside world, on Kubernetes. As a result, you are unlikely to find a prebuilt piece of software that does that -- nobody needs it. Management is done out-of-band through the API server. > unity You probably shouldn't rely on production containers to have debugging tools. The ideal container is "scratch" with your application binary in it, or distroless if that is not possible. It's not the container's job to have debugging tools -- it's the management layer's job to attach your favorite debugging tools to the pod while you're debugging. Support for this is pretty limited, though; k8s 1.16 got ephemeral containers, but they're still "alpha". > containers -in production- solve no problem They solve a variety of problems. You are guaranteed to get the code that you want into production, without any other things that you don't want. You get service discovery. You get declarative management. If you just have one thing you want to run, it's a lot of overhead. Traditionally management has been the job of the application itself. If it wants rate limiting, it adds rate limiting. If it wants log rotation, it adds log rotation. If it wants monitoring, it adds monitoring. If it wants authentication, it adds authentication. This is a waste of time for every application to have to implement these core features -- if the application knows that it's going to run inside infrastructure that already has that, it only has to focus on its core functionality. That means smaller, more reliable apps with less surface area for bugs or security problems. But it also means that to run it, you have to build the infrastructure. Right now, things are transitional -- apps exist on a spectrum of what services they expect to get from the operator and which services they provide themselves. It's a mess. But it will get better.

by berkes 6y ago

jrockway 6y ago

Containers aren't really designed to run without some sort of management infrastructure to handle the problems you mention -- network policy, monitoring, log rotation, scheduling, etc. There are many container runtimes. Kubernetes is popular and solves some of these problems. > Firewalling The base unit in Kubernetes is the Pod, which kind of acts as a mini-VM (without virtualization) that runs 1 or more containers. It's a mini-VM, so has its own network; 127.0.0.1 is the Pod, not the Node that it happens to run on. You can set many networking-related policies at the Pod level; PodSecurityPolicy for very coarse-grained security policies (access to the host network, access to become root, etc.), or NetworkPolicies for what people would traditionally consider firewall rules (only allow access to the MySQL pod from a pod in the MyWebApp service, etc). > Monitoring Kubernetes rotates your logs for you. Retaining logs and making them useful is still an unsolved problem; containerized or not. (I wrote a very complicated log analysis system when I was at Google. I miss it.) Your problems with monitoring (how to get Prometheus to discover containers to scrape) boil down to not having service discovery. Kubernetes provides service discovery, and Prometheus knows how to ask it for a set of endpoints to collect metrics from. (Service discovery is nice in general, and is often something sets of hand-rolled VPCs are missing.) > Timed Jobs Kubernetes has the concept of CronJobs. Overall, I don't like the approach of cron jobs over having a program that is always running and wakes up when it's time to do work. But both options are available to you. > fail2ban, tripwire, etc. Typically people have a broader-scoped access control and rate limiting policy. All external traffic hits a proxy between the outside network and the network that Pods are on. The proxy sends information about the request to an authorization service and receives an access decision, and accepts or rejects the request accordingly. If you had ssh sitting behind such a proxy, you would instruct it to send success/failure information to the authorization service so that it can make an accurate decision for the next connection attempt. There is some desire/work to make ACLs easier and do it at the NetworkPolicy/CNI layer. What you can do largely depends on what CNI plugin your cluster uses, and if you're using a managed k8s provider, you probably don't have much choice in the matter. Hence people replicating the "old world" with authenticating proxies. In general, people are not running SSH servers, especially ones that can accept passwords from the outside world, on Kubernetes. As a result, you are unlikely to find a prebuilt piece of software that does that -- nobody needs it. Management is done out-of-band through the API server. > unity You probably shouldn't rely on production containers to have debugging tools. The ideal container is "scratch" with your application binary in it, or distroless if that is not possible. It's not the container's job to have debugging tools -- it's the management layer's job to attach your favorite debugging tools to the pod while you're debugging. Support for this is pretty limited, though; k8s 1.16 got ephemeral containers, but they're still "alpha". > containers -in production- solve no problem They solve a variety of problems. You are guaranteed to get the code that you want into production, without any other things that you don't want. You get service discovery. You get declarative management. If you just have one thing you want to run, it's a lot of overhead. Traditionally management has been the job of the application itself. If it wants rate limiting, it adds rate limiting. If it wants log rotation, it adds log rotation. If it wants monitoring, it adds monitoring. If it wants authentication, it adds authentication. This is a waste of time for every application to have to implement these core features -- if the application knows that it's going to run inside infrastructure that already has that, it only has to focus on its core functionality. That means smaller, more reliable apps with less surface area for bugs or security problems. But it also means that to run it, you have to build the infrastructure. Right now, things are transitional -- apps exist on a spectrum of what services they expect to get from the operator and which services they provide themselves. It's a mess. But it will get better.