3 ms·
> but other than that, what's the difference? The need to configure, manage, monitor and maintain a whole extra set of programs. Also, systemd can just as easi
by falcolas 9y ago
> but other than that, what's the difference?
The need to configure, manage, monitor and maintain a whole extra set of programs. Also, systemd can just as easily start the binary in a cgroup, eliminating most of the following concerns.
> The isolation is broken
Worth noting that behind the scenes, Linux is using the same in-memory versions of libc, even for containers. That isolation you speak of already doesn't exist.
> [...] scheduling [...]
Ansible, Chef, Saltstack, etc.
> I have to manually manage each container port
You have 65,000 ports available to you. I don't think this is as big an issue as made out to be - people are lazy, which is the only reason there are so many conflicts over 8000 and 8443.
> Kubernetes, with enough hackery to work around bugs
Given the context that I am someone currently attempting that hackery - this shit just kinda works, some of the time. And when it fails, it's a morass of conflicting documentation, and responses of "well, you're not using the default distribution for your environment" (no shit, it's no longer supported) and "works on my single node minikube cluster". Give me ansible/chef/saltstack any day of the week. Hell, I'll even take cfengine at this point.
Kubernetes was built for Google's use case, in Google's environment. Attempting to use it with other restrictions and requirements is a nightmare.
- jchw 9y ago>The need to configure, manage, monitor and maintain a whole extra set of programs. Also, systemd can just as easily start the binary in a cgroup, eliminating most of the following concerns. In the former case, I need to start a (for example) Ubuntu image on my machine or VM. In the latter case, I need to start a Kubernetes image. It's not as different as you think. Kops can do the heavy lifting and even generate terraform manifests, which will put you far ahead of the manual solution in terms of controlling your infra. Also, Cgroups alone aren't enough, and is only a small part of Docker's isolation, although I did actually acknowledge that Systemd supports them... >Worth noting that behind the scenes, Linux is using the same in-memory versions of libc, even for containers. That isolation you speak of already doesn't exist. No it doesn't. Assuming elf binaries, the elf binfmt will call into ld-linux shared object when you hit a dynamic binary, which then does the runtime linking that links libc dynamically. You can confirm that this will call the ld-linux from the container when in a container. Compare how printf acts inside of Alpine containers versus on its Debian host. Or try hosting a CGO_ENABLED=1 Go binary in a container with no libc - it won't load because the kernel won't find the linker. If you put it in an Alpine container and it was built on a Ubuntu host, you might find it cryptically fails to load! >Ansible, Chef, Saltstack, etc. This is not scheduling. That is automation, configuration. SaltStack does not calculate how much CPU and RAM is available on a box to balance applications that are running. Before I used Kubernetes, I used SaltStack to manage fat Go binaries and even Docker containers for a moment, so I have a pretty good idea of what the differences are. >You have 65,000 ports available to you. I don't think this is as big an issue as made out to be - people are lazy, which is the only reason there are so many conflicts over 8000 and 8443. Nothing to do with laziness. Why would I want to have to pick a new pprof port every single time then remember what app has which? Why would I want to then manage a firewall for all of that? If you're doing this all by hand eventually you're going to screw up. I'd rather have isolated networking deny-by-default. >Given the context that I am someone currently attempting that hackery - this shit just kinda works, some of the time. And when it fails, it's a morass of conflicting documentation, and responses of "well, you're not using the default distribution for your environment" (no shit, it's no longer supported) and "works on my single node minikube cluster". Give me ansible/chef/saltstack any day of the week. Hell, I'll even take cfengine at this point. That is an absolutely inaccurate characterization of the very professional and helpful Kubernetes community. I've only had a few run-ins with them, but the experiences were absolutely completely the opposite and I've literally never heard "works on my single node minikube cluster" and I doubt you have either. >Kubernetes was built for Google's use case, in Google's environment. Attempting to use it with other restrictions and requirements is a nightmare. And finally, no it wasn't. It was built by the Google Cloud Platform team to help Google Cloud Platform users better utilize their VMs. There's a pretty detailed history that's all publicly available. As far as we know, Google is yet to use Kubernetes itself, other than their spin-off using it for Pokemon Go. Google internally uses some progression of Borg.
- markbnj 9y ago> Kubernetes was built for Google's use case, in Google's environment. Attempting to use it with other restrictions and requirements is a nightmare. That's a tremendous overstatement. The first part is just wrong, as was pointed out in earlier responses. The second part just smacks of your having been soured by a poor experience. We started with kubernetes in production at very small scale, just a few supporting services to begin with since there was a lot to learn. Currently we're running five or six of our core production apps on the GKE version of the platform and the experience has been very positive.
- deleted 9y ago[deleted]
- pmoriarty 9y ago"Given the context that I am someone currently attempting that hackery - this shit just kinda works, some of the time. And when it fails, it's a morass of conflicting documentation, and responses of "well, you're not using the default distribution for your environment" (no shit, it's no longer supported) and "works on my single node minikube cluster"." You could make pretty much the exact same set of complaints against all those configuration management tools (ansible/chef/salt/cfengine/puppet). They're all a huge mess of spaghetti and hackery that works when they work and can be a nightmare otherwise. All these tools need at least a couple of decades more to mature.
- funnelsgun 9y agoWe moved our infrastructure from scheduling containers with configuration management to scheduling containers with Nomad... and it is so much better. We no longer have the concern of defining static ports for each app, configuring App A with the port for App B, autoscaling when traffic increases 3x as someone attempts a L7 DDOS... I could go on!