13 ms·
Containers vs. Zones vs. Jails vs. VMs
- qaq 10y agoSmartOS run containers in zones get the best of both worlds
- johnramsden 10y agoI always run my containers in a jail which I run in a zone which is running in a VM just in case.
- mcnaughtonrules 10y agoAll behind NAT to make it securr
- erikb 10y agoAnd never connect it to the internet, ever. Island is best land.
- qaq 10y agoI appreciate the humor but having an ability to run containers in a zone gives you very good isolation without overhead of a VM.
- peterwwillis 10y agoSHOUTING IN TEXT DOESN'T HELP
- elastic_church 10y agoemphasis added
- cperciva 10y agoThe meme image ("Can't have 0days or bugs... if I don't write any code") is incorrect. You can't have bugs if you don't have any code, but not writing code just means that your bugs are guaranteed to be someone else's bugs. Now, this may be a good thing -- other people's code has probably been reviewed more closely than yours, for one thing -- but using other people's code doesn't make you invulnerable, and other people's code often doesn't necessarily match your precise requirements. If you have a choice between writing 10 lines of code or reusing 100,000 lines of someone else's code, unless you're a truly awful coder you'll end up with fewer bugs if you take the "10 lines of code" option.
- geofft 10y agoThere's probably no good way to pick up this context from the article, but the meaning of that particular meme is that the caption is supposed to be a shortsighted analysis. See http://knowyourmeme.com/memes/roll-safe http://knowyourmeme.com/memes/roll-safe , which lists examples like "You can't be broke if you don't check your bank account" or "If you're already late.. Take your time.. You can't be late twice."
- rileymat2 10y ago> If you have a choice between writing 10 lines of code or reusing 100,000 lines of someone else's code, unless you're a truly awful coder you'll end up with fewer bugs if you take the "10 lines of code" option. I disagree, this is only true if you understand why the other code has 100k lines [Although this example is a bit extreme]. A good example that could send a junior developer astray is date handling. Or most likely date mishandling if they are coding it themselves.
- cperciva 10y agoSure. I'm talking about the case where the 100k lines of code provides a large set of features you're not intending to use.
- jjn2009 10y agoThese container and container like solutions are not 10 lines of code, no implementation will be 10 lines. Therefore solutions which have had time to stabilize will be better since 10 lines of code isn't even a valid solution. New code causes new issues and increased complexity, thats the only point to be made by the meme.
- swordswinger12 10y agoI'm not an OS person, so forgive me if this is a stupid question: Lots of people are excited about Intel SGX and similar things. Are there any interesting ways people are thinking about combining, like, Docker containers with SGX enclaves and such? One could imagine (e.g.) using remote attestation to verify an entire container image.
- geofft 10y agoIt's definitely not a stupid question. I think you can't do it very well because a contained process is just a process on the host system, albeit with a few things changed (like what network devices it sees), and it's pretty hard to make an enclave as big as an entire OS process. However, see VMware's Overshadow paper for a pretty clever system that lets you run a verified process inside an untrusted kernel instide a trusted hypervisor: https://labs.vmware.com/academic/publications/overshadow https://labs.vmware.com/academic/publications/overshadow You might be able to do something similar.
- wmf 10y agoYeah, it's called SCONE: https://www.usenix.org/conference/osdi16/technical-sessions/presentation/arnautov https://www.usenix.org/conference/osdi16/technical-sessions/... It's pretty kludgey due to SGX limitations like not supporting fork().
- rickhanlonii 10y agoGreat rant about something way over my head by someone who knows way more than me about it! If I can transfer to a domain I understand better (front-end dev): It sounds like VMs, Jails, and Zones are like Ember.js: it comes with everything built in and is simple if you stay within the design. Containers are more like React: it gives you the pieces to build it yourself, and building it all yourself can lead to complexity, bugs, and performance issues. Disclosure: I have no idea what I'm talking about
- erikb 10y agoThanks for sharing what people see here that usually don't work in the backend.
- ianburrell 10y agoI think that the important feature of containers is that they are running from image containing the application. Zones, jails, and VMs are other isolation mechanisms that could be used to run containers. Running an application image unpacked into VM would be container. One place where the difference in namespace is visible is with Kubernetes pods. Containers running in pod share network and volumes namespaces.
- mrunalp 10y agoThe kubernetes pods share ipc and network namespaces (and pid soon). You may be referring to mount namespace by volumes namespace. The mount namespace is the minimum that distinguishes one container from the other as that's the mean by which they see different file-system hierarchies created from their images. k8s volumes are bind mounted into the containers in a pod.
- ianburrell 10y agoI wasn't sure if pods shared mount namespace or just the external volumes. It sounds like the same external volumes are mounted in each container but the rest of mounts are different.
- lloydde 10y agoYour definitions are not the common ones. Dropping the Linux from the full name Linux Containers doesn't make it a different fundamental. Containers, Zones and Jails are all OS layer virtualisation :https://en.m.wikipedia.org/wiki/Operating-system-level_virtualization https://en.m.wikipedia.org/wiki/Operating-system-level_virtu... . The style of application packaging that came with Docker compliments OS virt but the prior art demonstrates that it is its own innovation. "Docker will do to apt what apt did to tar" @bcantril
- ianburrell 10y agoThere are Windows Server containers. Docker runs on FreeBSD using jails and Solaris using zones. Hyper runs containers on hypervisors and RancherVM on KVM. Container now has two meanings: the isolated application, and the container-based virtualization on Linux. The latter is the original usage (LXC is older than Docker). But the important innovation of Docker is combining containers with image distribution. The isolated application is more general then virtual machine so the concept is extended to other virtualization techniques.
- betaby 10y agoSome point are just wrong. Containers and jails have many design similarities which were dismissed by author. Notably PIDs, both containers and jails are nearly identical with regard, you can kind of have one leg here another there, although that harder to achieve with FreeBSD jails; both implementations do not hide PIDs from the host systems. Networking - jails can run on top of non-virtualazed IP/net dev, containers can run in such modes as well. Link is someones rant without tech details.
- felixgallo 10y agoYeah. I feel like the author could really have used a little more time in the industry, really maybe gotten to know about the subject matter before writing a blog post. If only she had spent some time working at a container company, or maybe studied up to work at one of the big tech firms. What a missed opportunity to be more technical.
- jzelinskie 10y agoI'm going to be captain obvious here and let people know that the post I'm replying to is being extremely sarcastic. The author of the article is extremely well known in the Linux container community.
- hueving 10y agoIt's really strange to reply sarcastically to someone pointing out the author made some fundamental mistakes. An appeal to authority in the form of a sarcastic reply really adds nothing to the discussion.
- xelxebar 10y agoAdding to what discussion, exactly? To my eyes most comments so far seem to be little more than "I agree with the author" or "I disagree". OP's comment on the other hand at least told me something about the person behind the article. That said. I am now curious about comparisons between containers/jails/etc across various different metrics that people care about. Also, what cool non-containerization uses of cgroups and namespaces have some of you gotten up to?
- bingo_cannon 10y agoI have been learning about containers fairly recently. What are these security vulnerabilities that the post talks about? I haven't come across any docs that mention security yet.
- betaby 10y agoEarlier implementations had bugs like this http://blog.bofh.it/debian/id_413 http://blog.bofh.it/debian/id_413
- rhizome 10y agoRoot escape: https://blog.paranoidsoftware.com/dirty-cow-cve-2016-5195-docker-container-escape/ https://blog.paranoidsoftware.com/dirty-cow-cve-2016-5195-do...
- hueving 10y agoIt's mainly an attack surface issue. A process running in a Linux container is just a process running in a regular OS with some extra bells and whistles for resource constraints and isolation. So when it comes to making a kernel call, it's a call directly to the same kernel all of the other containers are calling. This means the entire kernel/userspace API is the attack surface for a malicious container. Compare that to a VM where the attack surface is the API the hypervisor exposes to a virtual machine. It's not that the former is necessarily smaller, it's just that the modus operandi in systems administration has always been that if a person executes malicious code as a user on the OS, you better wipe the system because kernel vulnerabilities aren't treated with the same severity of hypervisor vulnerabilities. This is the reason your containers are actually executed in a dedicated VM if you use something like GCE.
- GalacticDomin8r 10y agoMore gibberish. The distinct lack of technical assertions is always a good give-away. Closest the author comes is maybe this: "Sharing X socket". It isn't clear to me if the author thinks this is exclusive to "containers" but it appears that way. Dead wrong in respect to at least Jails, I can't speak on Zones.
- opcenter 10y agoI had to read the post twice before I really got what she was saying. I think the distinction I would make is that while there are many more use cases that you can apply to Containers that may not apply to Jails, Zones, or VMs the most common use case of "run an app inside a pre-built environment" applies to all of them. Since I believe most users (or potential users) of Containers are only looking at that use case, it's harder to see the differences between the different technologies. My only hope is that anyone in a position of making a decision on which technology to use can at least explain at a high level the difference between a Container and a VM.
- nikcub 10y agoIts probably a good time to stop using containers to mean LXC considering the new OCI runc specs containers on Solaris using Zones and Windows using Hyper-V: https://github.com/opencontainers/runtime-spec/blob/master/spec.md https://github.com/opencontainers/runtime-spec/blob/master/s...
- ffk 10y agoI don't think anyone in the container dev community thinks that containers means LXC only. Even back in 2013, docker's front end api was designed to support other runtimes such as VMs and chroot. Perhaps this is a marketing story gone awry?
- erikb 10y agoAfaik even the docker noob tutorial already points out that it is not just LXC only.
- cyphar 10y agoDocker hasn't support LXC as a backend for at least a year.
- erikb 10y agoYou mean they replaced it? May also what I've read. Unsure.
- cyphar 10y agoThey've been using libcontainer and then moved to runC (which is effectively a standard-compliant wrapper around libcontainer) since before 1.0. LXC was only used in the early history of Docker, and it was pretty bad to be quite honest (it's better now but there's no chance Docker will switch back).
- lgas 10y agoIt's probably a good idea time to stop thinking that anyone cares what Solaris calls anything.
- brotherjerky 10y agoAs a novice, this was a great informative read. More posts like this on the Internet, please!
- patrickg_zill 10y agoIn earlier versions of ProxMox the openvz vms were called containers and the KVM vms were called vms. So it is pretty confusing overall. For myself I would point out that Zones, Jails, OpenVZ and LXC , even KVM, all pretend that they are fully separate from the host node OS. While Docker et al do not pretend this; in fact if you are running Apache on your host system and try to run a Dockerized web server on port 80 the Docker container might refuse to start. The other methods mentioned, can't even determine what they are running under.
- betaby 10y ago> pretend that they are fully separate from the host node OS No! You can see all processes of the OpenVZ and jails from the host system >in fact if you are running Apache on your host system and try to run a Dockerized web server on port 80 the Docker container might refuse to start depends on network options.
- patrickg_zill 10y agoSorry for not being clear ... the OpenVZ, LXC, Zones (not sure about Jails) all have their own startup options, and if they don't start "init" as the first process inside, then they have a method that fakes it. You can install random new packages of software and run them, from inside the "VM" - from what I know, you can't do this from within a container. FWIW using KVM all you can see from the host node (unless you use debugging tools) is the PID and related process info of the KVM instance (one process per VM you have started).
- cthalupa 10y ago>- from what I know, you can't do this from within a container. You most certainly can.
- AstralStorm 10y agoYou can easily run top inside Xen VM via xl console. Or via SSH. Manage it with a stack like salt or ansible. This is a super weak argument.
- nisa 10y agoAs a lowly user¹: linux containers are more like gaffer tape around namespaces and cgroups than something like lego. You want real memory usage in your cgroup? let's mount some fuse filesystem: https://github.com/lxc/lxcfs https://github.com/lxc/lxcfs - https://www.cvedetails.com/vulnerability-list/vendor_id-4781/product_id-32696/Canonical-Lxcfs.html https://www.cvedetails.com/vulnerability-list/vendor_id-4781... We have to gaffer tape with AppArmor and SELinux to fix all the holes the kernel doesn't care about: https://github.com/lxc/lxc/blob/master/config/apparmor/container-rules https://github.com/lxc/lxc/blob/master/config/apparmor/conta... Solaris Zones are more designed and an evolution from FreeBSD Jails. Okay, the military likely paid for that: https://blogs.oracle.com/darren/entry/overview_of_solaris_kernel_zones https://blogs.oracle.com/darren/entry/overview_of_solaris_ke... Maybe it's Deathstar vs. Lego. But I assume you can survive a lot longer in a Deathstar in vacuum than in your Lego spaceship hardened by gaffa tape. 1: I have uttermost respect for anyone working on this stuff. No offense, but as a user sometimes a lack of design and implementation of bigger concepts (not as in more code, but better design, more secure) in the Linux world is sad. It's probably the only way to move forward but you could read on @grsecurity Twitter years ago that this idea is going to be a fun ride full of security bugs. There might be a better way?
- djsumdog 10y agoI really wish this post went into more detail. It feels too high level to be useful. I ran into the memory issue recently. In DC/OS when you use the marathon scheduler, if you go above the allocated memory limit, the scheduler kills your task and restarts it. The trouble is, if you ran top inside your container, and you're running on a DC/OS node with 32GB of memory, top reports all 32GB of memory. So interpreters that do garbage collection (like Java) will just continue to eat memory if you don't specify the right limits/parameters. The OS will eve let it allocate past the container limit, but just kill the container afterwards. Now the container limit is available under the /proc/cgroups somewhere, but now interpreters need to check to see if they're running in a container and adjust everything accordingly. Of course you could always tell your scheduler not to hard kill something when it goes over a memory limit, which is why we never ran into that when we were running things on CoreOS since we didn't configure hard limits per container.
- lloydde 10y agoIt feels like Ms Frazelle's essay ends abruptly. I was looking forward to the other use cases of non-Linux containers. I think most people are considering these OS-level virtualization systems for the same or or very similar use cases: familiar, scalable, performant and maintainable general purpose computing. Linux containers win because Linux won. Linux didn't have to be designed for OS virt. People have been patient as long as they've continued to see progress -- and be able to rely on hardware virt. Containers are a great example of where even with all of the diverse stakeholders of Linux, the community continues to be adaptive and create a better and better system at a consistent pace in and around the kernel. That my $job - 2, Joyent, re-booted Lx-branded zones to make Linux applications run on illumos (descendent of OpenSolaris) is more than a "can't beat them join them strategy" as it allows their Triton (OSS) users full access, not only to Linux API and toolchains, but to the Docker APIs and image ecosystem and has been an environment for their own continued participation in micro services evolution. Although Joyent adds an additional flavor, it targets the same scalable, performant and maintainable cloud/IaaS/PaaS-ish use case. In hindsight, it's crazy that I worked at three companies in a row in this space, Piston Cloud, Joyent, Apcera, and each time I didn't think I'd be competing against my former company, but each time the business models as a result of the ecosystems shifted. Thankfully with $job I'm now a consumer of all of the awesome innovations in this space.
- erikb 10y agoI also felt that way. It it like the beginning of an awesome blog post. Maybe she'll continue later on after thinking more about it.
- dom0 10y agoI think an interesting bit here is that e.g. Solaris first had Zones (i.e. "Containers"), while virtualisation was added later (sun4v), while the story is exactly the other way around for Linux.
- jtchang 10y agoI don't understand why anyone would say to give up containers and just use Zones or VMs? Containers are solving a very real problem. The problem is that containers weren't as well marketed before Docker (or as user friendly).
- betaby 10y agoContainers were incredibly popular in the hosting industry for a long time https://en.wikipedia.org/wiki/Virtuozzo_(company) https://en.wikipedia.org/wiki/Virtuozzo_(company) at that time SWsoft and their Virtuozzo hosting platform Docker marketed container idea to the different audience.
- solidsnack9000 10y agoMany people's use case for containers is more or less identical to that of Zones/VMs -- they aren't sharing anything across containers.
- cthalupa 10y agoThe majority of use cases for containers seem to be as more of a deployment and/or scheduling solution and less of an overlap with the use cases for virtualization.
- hueving 10y agoBecause the vast majority of use cases are already solved by zones/VMs.
- dreamcompiler 10y agoIgnorance admission time: I still have no idea what problem containers are supposed to solve. I understand VMs. I understand chroot. I understand SELinux. Hell, I even understand monads a little bit. But I have no idea what containers do or why I should care. And I've tried.
- betaby 10y agoIncrease server utilization by packing multiple non-hostile tenants on it, quickly create test environments, have a volatile env. You can have all of those with VMs although at much higher CPU, RAM usage cost.
- boris 10y agoWith one big limitation: they must all run the same os kernel (so you cannot run say a Windows or FreeBSD container on a Linux host). In fact, nobody guarantees that say Fedora will run on an Ubuntu-built kernel. Or even on a kernel from a different version of Fedora. So, IMO, anything other than running the exact same OS on host and in container is a hack.
- the8472 10y ago> In fact, nobody guarantees that say Fedora will run on an Ubuntu-built kernel. "nobody guarantees" just means that you can't externalize the work of trying it and seeing if it works. I don't think that's a huge loss, considering the space of all possible kernels, configuration switches, patches and distro packages is huge. It's like refusing to use a hammer because nobody can assure you that hammer A was thoroughly tested with nail type B.
- boris 10y agoNo, its like using a nailgun A with nails Y when it's only guaranteed to work with nails X. Or like using a chainsaw A with chain Y when it says you should use X. But hey, at least you are not trying to use nails on a chainsaw... ;-)
- solarengineer 10y agoAlso see http://dtrace.org/blogs/brendan/2013/01/11/virtualization-performance-zones-kvm-xen/ http://dtrace.org/blogs/brendan/2013/01/11/virtualization-pe...
- irrational 10y agoWhere does Docker fit into all of this? Asking for a friend.
- deathanatos 10y agoWhile I'm not sure I agree entirely with the "Complexity == Bugs" section, the main point, that containers are first-class citizens but a (useful) combination of independent mechanisms is spot-on. This has real repercussions: most people I've spoken do don't know these things exist. They know containers do, they have a very vague idea what containers are, but they have no fundamental understanding of the underlying concepts. (And who can blame them? Really, it was marketed that way.) For example, pid_namespaces, and subreapers are an awesome feature¹, and are extremely handy if you have a daemon that needs to keep track of a set of child jobs that may or may not be well behaved. pid_namespaces ensure that if something bad happens to the parent, the children are terminated; they don't ignorantly continue executing after being reparented to init. Subreapers (if a parent dies, reparent the children to this process, not init) solve the problem of grandchildren getting orphaned to init if the parent dies. Both excellent features for managing subtrees of processes, which is why they're useful for containers. Just not only containers. But developers aren't going to take advantage of syscalls they have no idea that they exist, of course. ¹although I wish someone could tell me why pid_namespaces are root-only: what's the security risk of allowing unprivileged users to create pid_namespaces?
- chatmasta 10y agoThis is definitely true, but only as long as docker (or $container_runtime) remains lightweight enough that you can still use those independent parts on their own, compatibly with docker. The risk is that docker grows in complexity such that it creates new dependencies between these independent parts and therefore handicaps their power when used individually. As an example, it's easy to create network namespaces and add routing rules, interfaces, packet forwarding logic, etc all by using `ip netns exec`. But there is no easy way to launch a docker container into an existing netns. You need to use docker's own network tooling or build your own network driver, which may be more complex than what you need. This strikes me as a code smell in docker.
- josteink 10y ago> This is definitely true, but only as long as docker (or $container_runtime) remains lightweight enough that you can still use those independent parts on their own, compatibly with docker. As someone who has exclusively used LXC containers (which Docker is/was initially built on), none of this applies to me. Your issue is with Docker, the implementation, not containers as a concept. Sometimes I feel HN needs to get its head out of Dockers butt and see that there's a world out here too. How many people here even know there are other container types at all? I'm often inclined to think none. No really. Just once try real, raw containers without all that docker wrapping bloat. How everything works and is tied together is just clear as day and obvious at once. It's all simple and super refreshing.
- jo909 10y agoI think it's important to realize that the reduced isolation of containers can also have pretty significant upsides. For example monitoring the host and all running containers and all future containers only means running one extra (privileged) container on each host. I don't need to modify the host itself, or any of the other containers, and no matter who builds the containers my monitoring will always work the same. The same goes for logging. Mainly there is an agreed-upon standard that containers should just log to stdout/stderr, which makes it very flexible to process the logs however you want on the host. But also if your application uses a log file somewhere inside the container, I can start another container (often called "sidecar") with my tools that can have access to that file and pipe it into my logging infrastructure. If I want multiple containers can share the same network namespace. So I listen on "localhost:8080" in one container, and connect to "localhost:8080" in another, and that just works without any overhead. I can share socket files just the same. I can run one (privileged) container on each host that starts more containers and bootstraps f.e. a whole kubernetes cluster with many more components. You can save yourself much "infrastructure" stuff with containers, because the host provides them or they are done conceptually different. For example ntp, ssh, cron, syslog, monitoring, configuration management, security updates, dhcp/dns, network access to internal or external services like package repositories. My main point is that by embracing what containers are and using that to your advantage, you gain much more than by just viewing them as lightweight virtualisation with lower overhead and a nicer image distribution. Edit: I want to add that not all of that is necessarily exclusive to containers or mandatory. For example throwing away the whole VM and booting a new one for rolling updates is done a lot, but with containers it became a very integral and universally accepted standard workflow and way of thinking, and you will get looked at funny if you DON'T do it that way.
- Kiro 10y agoI'm trying to understand something. At my last work we had a big problem with "works for me". We started using Vagrant and all those problems disappeared. Then Docker became popular and all of a sudden people wanted to use that instead. But is Docker really suitable for this? While each Vagrant instance is exactly the same Docker runs on the host system. It feels like it will be prone to all sorts of dissimilarities.
- staticfloat 10y agoIt depends on what you want to keep constant between hosts; for many (dare I say most?) projects, docker will provide a sufficiently identical environment that it "just works", because the filesystem as seen by the same image on two different computers will be exactly the same. This is sufficient for many, many projects, and is the primary source of "works for me" problems, in my experience. However, if your application requires things like the CPU architecture set is the same, or the amount of memory available to the process is the same, etc... then no, docker will not give you this level of "sameness". I have had bugs that manifested on one docker host and not another because one host had certain x86 extensions available (AVX, or something similar) and the other didn't, and this caused a certain codepath to be followed. However this is probably extremely rare for non-performance-targeted code, as most projects will compile for a conservative subset of x86_64 instructions so that they can run everywhere. Anecdotally, I find learning the "Docker way" has transformed my development paradigm in a much more fundamental way than Vagrant ever did, mostly because docker containers are almost instantaneous to start, which enables their use for a whole range of applications that I would never have even considered for something like Vagrant. Because I can launch a docker container in ~300ms, I can use containers like native applications. I do everything from building software to compiling latex documents using docker just so that I don't have to worry about installing toolchains or configuring things just right; I get an image that does what I want, and then I invoke the docker containers as if I were invoking the actual tools themselves. Docker is just a tool, but it's a darned powerful one, and it's pretty fun to see how it continues to evolve right now. I highly suggest you check it out.
- erikb 10y agoIn my experience Vagrant only solves 50% of the problem. In complexity it's not really reproducable enough. Docker until now always did what I asked from it. So I'm confident that it at least solves 75% of the problem. I can suggest to fully move to docker.
- jlebrech 10y ago"Lego" it's "Lego"
- jlebrech 10y agobefore i even heard of containers i found out about jails and wondered if you could serve each users jail using an nginx config file in their home directory.
- twic 10y ago> Legos "LEGO is always an adjective. So LEGO bricks, LEGO elements, LEGO sets, etc. Never, ever "legos."" [1] The other one that gets me is "math". I know it's not really plural, but "mathematics" has an a on the end, so it's "maths!"! Or do Americans say "stat" for "statistics" as well? https://twitter.com/lego_group/status/842115345280294912 https://twitter.com/lego_group/status/842115345280294912
- teapowered 10y agoIf you have two red lego bricks and two yellow lego bricks, do you not have two reds and two yellows? You definitely have A couple of bricks of each colour.
- tannhaeuser 10y agoA couple of observations from someone not-so-familiar with containers: If the consensus is that containers for the most part are just a way to ship and manage packages along with their dependencies to ease library and host OS dependencies, I'm missing a discussion about container runtimes themselves being a dependency. For example, Docker has a quarterly release cadence I believe. So when your goal was to become independent of OS and library versions, you're now dependent on Docker versions, aren't you? If your goal as IT manager is to reduce long-term maintainance cost and have the result of an internally developed project run on Docker without having to do a deep dive into the project long after the project has been completed, then you may find yourself still not being able to run older Docker images because the host OS/kernel and Docker has evolved since the project was completed. If that's the case, the dependency isolation that Docker provides might prove insufficient for this use case. Another point: if your goal is to leverage the Docker ecosystem to ultimately save ops costs, managing Docker image landscapes with eg. kubernetes (or to a lesser degree Mesos) might prove extremely costly after all since these setups can turn out to be extremely complex, and absolutely require expert knowledge in container tech across your ops staff, and are also evolving quickly at the same time. Another problem and weak point of Docker might be identity management for internally used apps; eg. containers don't isolate Unix/Linux user/group IDs and permissions, but take away resolution mechanisms like (in the simplest case) /etc/password and /etc/group or PAM/LDAP. Hence you routinely need complex replacements for it, adding to the previous point.
- ProductsHair 10y agoWith that out of the way, I’d like to share some of the traits and characteristics that you can expect from me and my team as we build out our hair loss cure website for everyone’s benefit. http://hairlosstreatmentsreview.net/ http://hairlosstreatmentsreview.net/
- benmmurphy 10y agoI think jess is on the money here. The complexity in linux containers vs zones show up in two ways: 1) the linux kernel container primitives are implemented in ways that are more complicated. for example in zones pid separation is implemented by just checking the zone_id and if the zone_id is different then processes can't access each other. this also means that in zones pids are unique and you can't have two processes from two different zones with the same pid [with the exception: i believe they may have hacked something in to handle pid1 on linux]. similarly, in zones there is no user mapping if you are root inside the zone you are also root outside of the zone. the files you create inside a zone are uid: 0 and also uid: 0 outside the zone. if you look at how device permission is handled in linux we have cgroups that controls what devices can be accessed and created. while in solaris zones they use the existing Role Based Access Control and device visibility. so inside a zone you can either have permission to create all devices (very bad for security) or create no devices. In zones access to devices is mediated by whatever devices the administrator has created in your zone. in zones there is no mount namespace instead there is something that is very similar to chroot. it is just a vnode in your proc struct where you are restricted from going above. zones have mostly been implemented by just adding a zone_id to the process struct and leveraging features in solaris that already existed [i guess the big exception would be the network virtualization in solaris] while in linux there are all these complicated namespace things. this complexity means there are probably going to be more bugs in the linux kernel implementation. however, because you don't have as much fine grain control this can also create security bugs in your zone deployment. for example i found an issue in joyent's version of docker where you could trick the global zone into creating device files in your zone and these could be used to compromise the system. under a default lxc container this would not be possible because cgroups would prevent you from accessing the device even if you could trick someone else into creating it. you also have to be careful in zones with child zones getting access to files inside the parent zone. if you ever leak a filesystem fd or hard link into the child zone from the parent zone then all bets are off because the child is able to write into the parent zone as root. (i believe this situation was covered in a zone paper where they describe the risk of a non-privileged user in the global zone collaborating with a root user in a child zone to escalate privileges on the system) 2) because all the pieces are separate in linux then something has to put it together and make sure all the pieces are put together correctly. like i wouldn't trust sysadmins to do this on their own and luckily there are projects like lxc/lxd/docker etc that assemble these pieces in a secure way.
- AlexanderDhoore 10y agoNobody mentioned unikernels yet? It's a bit unrelated to the containers discussion in this thread, but I thought I'd mention it anyway. They let you create an operating system image, which only includes the code you need. Nothing more, nothing less. This improves security, because the attack surface is reduced. It makes a lot of sense too me when I think about how cloud computing works. Most of the time an operating system container, zone, jail, VM... is booted just to run a select number of processes. There is absolutely no need for a general purpose system. I think unikernels could really shine in this area. MirageOS is a project that lets you create unikernels. It's written in OCaml, so it's interesting in more than one way. MirageOS images mostly run on Xen, by the way. [1] https://en.wikipedia.org/wiki/Unikernel https://en.wikipedia.org/wiki/Unikernel [2] https://mirage.io/ https://mirage.io/
- newsat13 10y agoSo first step is containers as a service. And then this container can run as a unikernel as an implementation detail. Did I understand it correctly?
- chatmasta 10y agoI asked a similar question yesterday. [0] The problem is that containers share the kernel of the hostOS, so you cannot host a unikernel without some kind of hardware virtualization, since the unikernel is obviously a different kernel from that of the host OS. However you can run qemu inside docker if you want to inherit its sandboxing and namespace configurations. The problem comes when you have to isolate resources, like network devices, at both the namespace level on the host OS, and the virtualization level inside qemu. Intel's Clear Container project tries to solve this problem, but it's still limited by some virtualization overhead because qemu requires a tap device, which then connects to eth0 in a netns, which is one half of a veth pair with the host. So you end up creating 3 or 4 virtual Ethernet links just to route packets down to the guest. [0] https://news.ycombinator.com/item?id=13976125 https://news.ycombinator.com/item?id=13976125
- dom0 10y ago
- aaossa 10y agoA little bit off topic but I've been following Jess for a while and I think that developers like her are great. In my country is hard to see a happy developer and she seems to enjoy everything she does. That's why I follow her, because of her great work and great personality. I'm happy to see one of her blog posts here in HN
- kraemate 10y agoWe wrote a paper comparing containers and VMs for the middleware conference: http://people.cs.umass.edu/~prateeks/papers/a1-sharma.pdf http://people.cs.umass.edu/~prateeks/papers/a1-sharma.pdf
- HugoDaniel 10y agoIt doesn't matter how many distinction you make on these things (first-class, last-class, second-class, poor-class, etc...). These kind of discussions are always relative. All is good as long as your decision is conscious of the compromises taken by each approach and what they entail (what other security mechanisms do you have at your disposal ? how could they enhance your app ? will your solution depend on external tools like ansible/puppet/etc ? do you actually need "containers" or jails or [insert your favorite trendy tech here] ?). Running a *BSD or a Linux is a way bigger design decision than what kind of isolation mechanisms you have as many of the underlying parts are becoming different.
- fisholito 10y ago"container is not a real thing". But what could we say about real things within software field?
- throw2016 10y agoIt's a sad reflection of a technical community when 3 years later many do not seem to still clearly understand the bare basics of how containers work. HN has been complicit in massively hyping containers without a corresponding understanding of how containers work outside the context of docker. How many container users understand namespaces and how easy it is to launch a process in its own namespace, both as root and non root users? Or know overlay file systems and how they work. Or linux basics like bind mounts, and networking. The docker team leveraged LXC to grow from its tooling to container images but didn't shy from rubbishing it and misleading users on what it is. LXC was presented as 'some low level kernel layer' when it has always been a front end manager for containers like Docker, the only difference is LXC launches a process manager in the container and Docker doesn't. Just clearly articulating this in the beginning would have led to a much better understanding of containers and Docker itself among users and the wider community. How many docker users know the authors of aufs and overlayfs? The hype is so intense around the front end tools that few know or care to know the underlying tools. This has led to a complete lack of understanding of how things work and an unhealthy ecosystem as critical back end tools do not get funding and recognition, with the focus solely on front ends as they 'wrap' projects, make things more complex and build walls to justify their value. Launch 5000 nodes and 500000 containers. How many users need this? And this complexity has a huge cost and technical debt, when you are scaling as many stories here itself report and when you are trying to figure out the ecosystem so much so that its now at risk of putting people off containers. A stateless PAAS has never been the general use case, its a single use case pushed as a generic solution because that's Docker's origin as a PAAS provider. The whole problem with scaling for the vast majority is managing state. Running stateless containers or instances does not even begin to solve that in any remote way. Yes, it sounds good to launch 5000 stateless instances but how is it useful? Without state scaling has never been a problem. A few bash scripts which is what Dockerfiles are will do it. But now because of hype around Docker and Kubernetes users must deal with needless complexity around basic process management, networking, storage and re-architect their stack to make it stateless, without any tools to manage state. Congratulations on becoming a PAAS provider.
- floatboth 10y agoJails are actually very similar to Linux namespaces / unshare. Much more similar than most people in this thread think. There's one difference though: In namespaces, you start with no isolation, from zero, and you add whatever you want — mount, PID, network, hostname, user, IPC namespaces. In jails, you start with a reasonable secure baseline — processes, users, POSIX IPC and mounts are always isolated. But! You can isolate the filesystem root — or not (by specifying /). You can keep the host networking or restrict IP addresses or create a virtual interface. You can isolate SysV IPC (yay postgres!) — or keep the host IPC namespace, or ban IPC outright. See? The interesting parts are still flexible! Okay, not as flexible as "sharing PIDs with one jail and IPC with another", but still. So unlike namespaces, where user isolation is done with weird UID mapping ("uid 1 in the container is uid 1000001 outside") and PID isolation I don't even know how, jails are at their core just one more column in the process table. PID, UID, and now JID (Jail ID). (The host is JID 0.) No need for weird mappings, the system just takes JID into account when answering system calls. By the way, you definitely can run X11 apps in a jail :) Even with hardware accelerated graphics (just allow /dev/dri in your devfs ruleset). P.S. one area where Linux did something years before FreeBSD is resource accounting and limits (cgroups). FreeBSD's answer is simple and pleasant to use though: https://www.freebsd.org/cgi/man.cgi?rctl https://www.freebsd.org/cgi/man.cgi?rctl
- arca_vorago 10y agoAs a sysadmin I just want to point out to this mostly dev crowd, that my current favorite method of operations is to have multiple compartmentalized VM's which then may or may not hold containers or jails. Why do I do it this way? Because having a full stack VM for each use-case on a good server is realistically not that much more resource hungry than a container, but the benefits are noticeable. Lots of the core reason stems from security concerns. For example, there are quite a few Microsoft Small Business Server styled linux attempts at hitting the business space, but instead of playing to the strengths of modern hardware, they all mostly throw every service on the same OS just like SBS does... which is a major weakness. So instead of an AD server that also does dns and dhcp and the list goes on, each thing in my environments get it's own seperate VM (eg, SAMBA4 by itself, bind by itself, isc-kea by itself, and so on) Another reason for this is log parsing related. It's much easier to know that when the bind VM OSSEC logs go full alert, I know exactly what to fix. On multi container systems, a single failure or comprimise can end up affecting many containerizations and convoluting the problem/solution process. Of course, the main weakness to such a system is any attempt to break out of the VM space illicitly could comprimise many systems, but that's why you harden the VM's and have good logging in the first place, but also do it to the host system, along with using distributed seperation of hosts and good backups. Just some real world usage from a sysadmin I wanted to convey. I still will do a container or a VM with many containers for the devs if needed, but when it comes time to deploy to prod, I tend to use a full stack VM. I'm also open to talk about weaknesses in this system, as I'd be curious to hear what devs think. To be fair, I still haven't fully caught up with the whole devops movement either, so perhaps I'm behind. Also, a big shoutout to proxmox for a virtual environment system, FOSS and production quality since 4.0. I have also run BSD systems with jails in a similar way. The key pont of the article is that zones/jails/vms are top level isolations and containers are not (but that doesn't make containers bad!)
- shamsalmon 10y agoHave you considered a solution like graylog for centralized logging with both the VMs and Containers? Thats what my company tends to do with clients and we frequently choose containers and multiple VMs together running something like docker stack. As long as you do proper security practices on the container itself, such as not running the application as root, I dont see any downside to running things in containers over VMs as long as you have proper logging. Combined with something like ansible or another config management you can automate this whole process and it works really well for us.
- apeace 10y agoIn this post the author links to one of her previous posts[0], where she wrote: > As a proof of concept of unprivileged containers without cgroups I made binctr. Which spawned a mailing list thread for implementing this in runc/libcontainer. Aleksa Sarai has started on a few patches and this might actually be a reality pretty soon! Does anybody know if this made it into runc/libcontainer? I'm not an expert on these technologies but would love to read through docs if it has been implemented. [0] https://blog.jessfraz.com/post/getting-towards-real-sandbox-containers/ https://blog.jessfraz.com/post/getting-towards-real-sandbox-...
- zpallin 10y agoI found this article to be humorous and informative.
- cowardlydragon 10y ago"PERIOD." This is almost always the harbinger of lies, deception, propaganda, or lack of nuance.
- deavmi 10y agoI only came here for the meme.
- vor1968 10y agoIs it possible to design a process that isn't a self-contained OS instance depending on a lot of horseshit overlay controls to perform a task in a well designed way that still allows privilege separation from the host OS in a manageable way? Of course not. It's the basis for all this other shite.