6 ms·
They're killing performance (one of the main reasons to use containers) and adding a massive extra layer of management if they're running containers on EC2. I
by nailer 10y ago
They're killing performance (one of the main reasons to use containers) and adding a massive extra layer of management if they're running containers on EC2.
I suspect Netflix are too wedded to AWS (which is weird as Amazon is their biggest threat) but Triton or Red Shift (both of which actually isolate containers using SmartOS and SELinux respectively) make way more sense for other people who want to use the blazing fast IO speed of containers on bare metal.
- richardwhiuk 10y agoOr use something like GKE where the containers should be running directly on hardware, so you only have one layer.
- nailer 10y ago"Bare-metal" means running directly on hardware. But yes, GKE would also join Triton and Red Shift (provided it has similar container isolation capabilities).
- nailer 10y agoActually, as 0x74696d mentions below, GKE is also not bare metal. Doing a little research: GKE runs on https://cloud.google.com/container-optimized-os/docs/ https://cloud.google.com/container-optimized-os/docs/ which is designed for GCE, which runs on KVM: https://cloud.google.com/compute/docs/faq https://cloud.google.com/compute/docs/faq So it looks like just Triton and Red Shift. Disclaimer: I used to work at Red Hat, which means I like the people behind Red Shift, but that also means I hate the people behind Triton as part of the early 2000s Linux/Solaris wars.
- 0x74696d 10y agoGKE clusters run on VMs, not on bare metal.
- lima 10y agoContainers don't actually run directly on hardware with GCE - there's still a virtualization layer in-between. I'm 99% sure that Google runs one VM per container because that's the only way to make it safe. Anything else would be insane.
- jakon89 10y ago1 container per VM? What's the point? If you can bypass container sandbox it's very likely that you can do the same with VM.
- lima 10y ago> If you can bypass container sandbox it's very likely that you can do the same with VM. Hypervisors are much, much harder to break out of than a Linux container.
- outworlder 10y ago> I'm 99% sure that Google runs one VM per container I'm 100% sure you are wrong. You might as well just use VMs. Containers are not only about safety, you know.
- lima 10y agoI stand corrected, it's a bunch of VMs per customer but still no multi-tenancy.
- deleted 10y ago[deleted]
- retreatguru 10y agoIt seems they value developer productivity and innovation over machine performance. I find this enlightening.
- qeternity 10y agoCompute is cheap. People are expensive.
- dominotw 10y ago> Compute is cheap. Not on AWS.
- exhilaration 10y agoWord on Hacker News is that Netflix is one of AWS' premiere showcase customers and gets discounts that we couldn't dream of.
- djb_hackernews 10y agoIf you have dynamic workloads, need to be HA, scalable and multi region, AWS is worth every cent. If you are running a webapp and a database with a known workload, not so much.
- 1_2__3 10y agoThe refrain of shitty developers everywhere.
- randomthoughtss 10y agoSeems fine for server-side development. For client-side, please don't kill my battery.
- radicalbyte 10y agoPerformance has never mattered. That's why Internet Explorer's map access implementation was exponential instead of logarithmic (to the size of the map). That one piece of lazy coding held back Javascript on their platform for years, and I'd argue that it as the most significant cause of their browser's market share plummeting (or at last tied with lack of adblockers).
- jo909 10y agoCould you elaborate on why containers don't perform on EC2? I'm not running that combination myself so I wouldn't really know, but I'm not aware of problems with that specific combination or can think of anything obvious.
- throwasehasdwi 10y agoContainers use LXC in the Linux kernel underneath. Overhead is even lower than HVM virtualization which is a couple percent for most things. Containers are really just a smarter way of dividing resources between users on a shared linux box, something that's been going on since the dawn of time. It might add another 1% overhead for most tasks to run containers on HVM virtual machines. The one giant exception is network performance. The network is usually virtualized at the VM level, unless you have an "enhanced networking" VM with SR-IOV enabled. For containers it's virtualized a second time. This makes the combo potentially terrible if you're trying to run high bandwidth stuff on low performance VM's. I still like the combo because it allows you to give the big ole finger to AWS if they try to lock you in one day. Since your containers are isolated from the VM you can easily spin them up pretty much anywhere else.
- djb_hackernews 10y agoThere is a lot of incorrect information in this post. - Containers are a combination of namespaces, cgroups, and chroot (maybe). You don't need LXC to use containers. Docker doesn't even use LXC. - There is no overhead for running processes in containers. - There is no requirement to virtualize networks for containers. They can be configured to use the host's network directly, at which point you are bound by the host's network capabilities. Otherwise it is typically a combination of bridges and overlay networks for which the benefits outweigh the performance concerns for most workloads.
- throwasehasdwi 10y agoI was wrong about docker, back when I was playing with it it did use LXC, and appears to have started out as project to make a specialized version of LXC. You're right that Docker has its own container runtime now. The overhead for running containers is usually very low but real. The OS needs to partition low level resources that are normally shared and the scheduling introduces some overhead. I disagree about network performance. The virtualization adds a somewhat small but non-trivial overhead here (the overhead for other stuff could probably be considered trivial) Here is a paper I dug up on that gives results to back up my ranting. It's a bit old now but probably still holds mostly true. http://domino.research.ibm.com/library/cyberdig.nsf/papers/0929052195DD819C85257D2300681E7B/$File/rc25482.pdf http://domino.research.ibm.com/library/cyberdig.nsf/papers/0...
- lima 10y agoSELinux is NOT isolation. The main issue is the large kernel attack surface and SELinux, while important, only solves a small part of that.
- nailer 10y agoThat's true, but a lot of the classic containers-not-containing issues (/sysfs hacks to get into the parent kernel etc) are prevented by SELinux policies. See https://blog.openshift.com/securing-dockers-future-with-selinux-and-the-open-source-way/ https://blog.openshift.com/securing-dockers-future-with-seli...
- lima 10y agoAgreed. But people keep selling SELinux or AppArmor as a solution for multi-tenant container environments, which is just plain false. The real solution are efforts like like Intel's Clear Containers and Hyper's runV.
- nailer 10y agoLate edit: I've said 'Red Shift' above, I mean Red Hat's OpenShift. Tech has too many edgy code names.
- sleepybrett 10y ago.. so kubernetes.
- nailer 10y agoKubernetes on bare metal, with additional tech to isolate VMs from each other safely. See link elsewhere in thread.
- deleted 10y ago[deleted]
- brendangregg 10y agoWe get fast performance when we need to, and there's a number of EC2 technologies that help with that.