3 ms·
Containers aren't really designed to run without some sort of management infrastructure to handle the problems you mention -- network policy, monitoring, log ro
by jrockway 6y ago
Containers aren't really designed to run without some sort of management infrastructure to handle the problems you mention -- network policy, monitoring, log rotation, scheduling, etc.
There are many container runtimes. Kubernetes is popular and solves some of these problems.
> Firewalling
The base unit in Kubernetes is the Pod, which kind of acts as a mini-VM (without virtualization) that runs 1 or more containers. It's a mini-VM, so has its own network; 127.0.0.1 is the Pod, not the Node that it happens to run on. You can set many networking-related policies at the Pod level; PodSecurityPolicy for very coarse-grained security policies (access to the host network, access to become root, etc.), or NetworkPolicies for what people would traditionally consider firewall rules (only allow access to the MySQL pod from a pod in the MyWebApp service, etc).
> Monitoring
Kubernetes rotates your logs for you. Retaining logs and making them useful is still an unsolved problem; containerized or not. (I wrote a very complicated log analysis system when I was at Google. I miss it.) Your problems with monitoring (how to get Prometheus to discover containers to scrape) boil down to not having service discovery. Kubernetes provides service discovery, and Prometheus knows how to ask it for a set of endpoints to collect metrics from. (Service discovery is nice in general, and is often something sets of hand-rolled VPCs are missing.)
> Timed Jobs
Kubernetes has the concept of CronJobs. Overall, I don't like the approach of cron jobs over having a program that is always running and wakes up when it's time to do work. But both options are available to you.
> fail2ban, tripwire, etc.
Typically people have a broader-scoped access control and rate limiting policy. All external traffic hits a proxy between the outside network and the network that Pods are on. The proxy sends information about the request to an authorization service and receives an access decision, and accepts or rejects the request accordingly. If you had ssh sitting behind such a proxy, you would instruct it to send success/failure information to the authorization service so that it can make an accurate decision for the next connection attempt.
There is some desire/work to make ACLs easier and do it at the NetworkPolicy/CNI layer. What you can do largely depends on what CNI plugin your cluster uses, and if you're using a managed k8s provider, you probably don't have much choice in the matter. Hence people replicating the "old world" with authenticating proxies.
In general, people are not running SSH servers, especially ones that can accept passwords from the outside world, on Kubernetes. As a result, you are unlikely to find a prebuilt piece of software that does that -- nobody needs it. Management is done out-of-band through the API server.
> unity
You probably shouldn't rely on production containers to have debugging tools. The ideal container is "scratch" with your application binary in it, or distroless if that is not possible. It's not the container's job to have debugging tools -- it's the management layer's job to attach your favorite debugging tools to the pod while you're debugging. Support for this is pretty limited, though; k8s 1.16 got ephemeral containers, but they're still "alpha".
> containers -in production- solve no problem
They solve a variety of problems. You are guaranteed to get the code that you want into production, without any other things that you don't want. You get service discovery. You get declarative management. If you just have one thing you want to run, it's a lot of overhead.
Traditionally management has been the job of the application itself. If it wants rate limiting, it adds rate limiting. If it wants log rotation, it adds log rotation. If it wants monitoring, it adds monitoring. If it wants authentication, it adds authentication. This is a waste of time for every application to have to implement these core features -- if the application knows that it's going to run inside infrastructure that already has that, it only has to focus on its core functionality. That means smaller, more reliable apps with less surface area for bugs or security problems. But it also means that to run it, you have to build the infrastructure. Right now, things are transitional -- apps exist on a spectrum of what services they expect to get from the operator and which services they provide themselves. It's a mess. But it will get better.
- berkes 6y agoThanks for the eloquent reply! During my PoC I did see that a lot of things that I traditionally do on the machine are now done external. But for me that makes it harder. A practical example was: How do I get all my "fail2ban"-rules ported to "whatever this cluster-management uses"? Simple things like "if someone requests admin.php on this Rust-only API: ban it for 2 hours". I did see this is possible, but the amount of tools and their complexity was off-putting to me. I really appreciate how you suggest good alternatives and solutions to problems I stated: I hope they'll help me or someone else moving to k8s in future. And about: > Traditionally management has been the job of the application itself. Yes. Been there; hell, I even wrote rate-limiters for Drupal, and WordPress for our large customers on the Drupal, and later WordPress webhoster that I ran. But I also understand how microservices is the solution to this. However: Microservices does not require containerization, IMO. I'm running between 8 and 14 VMs on Linode and DigitalOcean exactly because of this: because stuff is "microserviced". I have a flock of Elasticsearch servers. A server that ingest, processes and then stores the OSM database weekly into that elasticsearch. Several postgres database servers. A statistics-server. Had an "avatar server", there's a dedicated url-shortener somewhere, A mailserver. Moving an authentication server onto my flock next week. I don't need k8s for that. Microservices are a good solution to specific problems but the underlying tech can easily be $6.99 VPSs managed with ansible.