12 ms·

Why did containers happen?

208 points by todsacerdoti 1y ago

https://www.youtube.com/watch?v=eMU2mZgo99c https://www.youtube.com/watch?v=eMU2mZgo99c

all2 1y ago
Likely because Plan9's 'everything-is-a-filesystem' failed.
- jauntywundrkind 1y ago
  Never grew popular, perhaps. But I'm not sure how it failed, and not sure how many of the Venm Diagrams of concerns plan9 really has with containers. Yes there was an idea of creating bespoke filesystems for apps, custom mount structures that plan9 had. That containers also did something semi-parallel to. But container images as read only overlays (with a final rw top overlay) feel like a very narrow craft. Plan9 had a lot more to it (everything as a file), and containers have a lot more to them (process, user, net namespaces, container images to pre-assembled layers). I can see some shared territory but these concerns feel mostly orthogonal. I could easily imagine a plan9 like entity arising amid the containerized world: these aren't really in tension with each other. There's also a decade and a half+ gap between Plan9's hayday and the rise of containers.
- walkabout 1y ago
  The standard answer is, "because inventing and implementing them was easier than fixing Python packaging."
  - aaroninsf 1y ago
    Ngl this is why I started using them
  - LexiMax 1y ago
    I think "fixing distro packaging" is more apropos. In a past life, I remember having to juggle third-party repositories in order to get very specific versions of various services, which resulted in more than few instances of hair-pull-inducing untangling of dependency weirdness. This might be controversial, but I personally think that distro repos being the assumed first resort of software distribution on Linux has done untold amounts of damage to the software ecosystem on Linux. Containers, alongside Flatpak and Steam, are thankfully undoing the damage.
    - bombcar 1y ago
      I wonder if it can be traced back to something RedHat did somewhere, because it may have all begun once you COULDN'T be absolutely certain that anything even remotely "enterprise" was running on a RedHat.
      - LexiMax 1y ago
        I think it's a natural outgrowth of what Linux is. Linux is just a kernel - you need to ship your own userland with it. Therefore, early distros had to assemble an entire OS around this newfangled kernel from bits and pieces, and those bits and pieces needed a way to be installed and removed at will. Eventually this installation mechanism gets scope creep and and suddenly things like FreeCiv and XBill are distributed using the same underlying system that bash and cron use. This system of distro packaging might be good as a selling point for a distro - so people can brag about their distro comes with 10,000 packages or whatever. That said, I can think of no other operating system out there where the happiest path of releasing software is to simply release a tarball of the source, hope a distro maintainer packages it for you, hope they do it properly, and hope that nobody runs into a bug due to a newer or older version of a dependency you didn't test against.
        skydhash 1y ago
        Yours is a philosophy I encounter more and more. Where there should be that unified platform, ideally fast moving, where software is only tested against $latest. Stability is a thing of the past. The important thing is more feature. Instead of designing a solution and perfecting it overtime, it's endless tweaking where there's a new redesign every years. And you're supposed to use the exact computer as the Dev to get their code to work.
      - ghaff 1y ago
        Red Hat was actually doing something more directly based on a variety of existing Linux projects than Docker but switched to OCI/Docker when that came about--rather than jumping on the CloudFoundry bandwagon. (Which many argues was obviously the future for container orchestration.) Kubernetes was also not the obvious winner in its time with Mesos in particular seeming like a possible alternative when it wasn't clear if orchestration and resource management weren't possibly different product categories. I was at Red Hat at the time and my impression was they did a pretty good job of jumping onto where the community momentum at the time was--while doubtless influencing that momentum at the time.
jmclnx 1y ago
FreeBSD jails years ago based upon a user request. >hosting provider's ... desire to establish a clean, clear-cut separation between their own services and those of their customers https://en.wikipedia.org/wiki/FreeBSD_jail https://en.wikipedia.org/wiki/FreeBSD_jail My guess Linux started getting requests rom various orgs for a while, so in true Linux fashion, we got a a few different container type methods years later. I still think Jails are the best of the bunch, but they can be a bit hard to setup. Once setup, Jails works great. So here we are :)
IlikeKitties 1y ago
I mean, containers do lend themselves to cargo culting by their very nature.
- NewJazz 1y ago
  https://youtube.com/watch?v=TwxetMVdYTc&t=30s https://youtube.com/watch?v=TwxetMVdYTc&t=30s
blu3h4t 1y ago
You can laugh or not but its because they never finished gnu/hurd :D
hedgehog 1y ago
Containers happened because running an ad network and search engine means serving a lot of traffic for as little cost as possible, and part of keeping the cost down is bin packing workloads onto homogeneous hardware as efficiently as possible. https://en.wikipedia.org/wiki/Cgroups https://en.wikipedia.org/wiki/Cgroups (arguably FreeBSD jails and various mainframe operating systems preceded Linux containers but not by that name)
- cbdumas 1y ago
  What does the 'ad network and search engine' have to do with it? Wouldn't any organization who serves lots of traffic have the same cost cutting goals you mentioned?
  - wmf 1y ago
    It's an oblique way to say that Linux cgroups and namespaces were developed by Google.
    - hedgehog 1y ago
      Yes, to expand: Both search and ads mean serving immense amounts of traffic and users while earning tiny amounts of revenue per unit of each. The dominant mid-90s model of buying racks of Sun and NetApp gear, writing big checks to Oracle, etc, would have been too expensive for Google. Instead they made a big investment in Linux running on large quantities of commodity x86 PC hardware, and building software on top of that to get the most out of it. That means things like combining workloads with different profiles onto the same servers, and cgroups kind of falls out of that. Other companies like Yahoo, Whatsapp, Netflix also followed interesting patterns of using strong understanding of how to be efficient on cheap hardware. Notably those three all were FreeBSD users at least in their early days.
      - dmoy 1y ago
        Yup and just to add timelines - Google Borg and containerization was what... 2003-2005? Docker was 2011-2013?
rshnotsecure 1y ago
Fascinating documentary on Kubernetes for those who have 50 minutes. Gives more background to the "Container Wars". The filmmakers also have documentaries on the history of Python, Argo, etc. Some highlights: - How far behind Kubernetes was at the time of launch. Docker Swarm was significantly more simple to use, and Apache Mesos scheduler could already handle 10,000 nodes (and was being used by Netflix). - RedHat's early contributions were key, despite having the semi-competing project of OpenShift. - The decision to Open Source K8S came down to one meeting brief meeting at Google. Many of the senior engineers attended remotely from Seattle, not bothering to fly out because they thought their request to go OS was going to get shutdown. - Brief part at the end where Kelsey Hightower talks about what he thinks might come after Kubernetes. He mentions, and I thought this was very interesting ... Serverless making a return. It really seemed like Serverless would be "the thing" in 2016-2017 but containers were too powerful. Maybe now with KNative or some future fusing of Container Orchestration + K8S? [1] - https://youtu.be/BE77h7dmoQU https://youtu.be/BE77h7dmoQU
- btreecat 1y ago
  I feel that's going to be more interesting than this video. The speaker is very unpracticed.
alphazard 1y ago
Containers (meaning Docker) happened because CGroups and namespaces were arcane and required lots of specialized knowledge to create what most of us can intuitively understand as a "sandbox". Cgroups and namespaces were added to Linux in an attempt to add security to a design (UNIX) which has a fundamentally poor approach to security (shared global namespace, users, etc.). It's really not going all that well, and I hope something like SEL4 can replace Linux for cloud server workloads eventually. Most applications use almost none of the Linux kernel's features. We could have very secure, high performance web servers, which get capabilities to the network stack as initial arguments, and don't have access to anything more. Drivers for virtual devices are simple, we don't need Linux's vast driver support for cloud VMs. We essentially need a virtual ethernet device driver for SEL4, a network stack that runs on SEL4, and a simple init process that loads the network stack with capabilities for the network device, and loads the application with a capability to the network stack. Make building an image for that as easy as compiling a binary, and you could eliminate maybe 10s of millions of lines of complexity from the deployment of most server applications. No Linux, no docker. Because SEL4 is actually well designed, you can run a sub kernel as a process on SEL4 relatively easily. Tada, now you can get rid of K8s too.
- tptacek 1y ago
  This makes sense if you look at containers as simply a means to an end of setting up a sandbox, but not really much sense at all if you think of containers as a way to make it easy to get an arbitrary application up and running on an arbitrary server without altering host system dependencies.
  - ianburrell 1y ago
    I suspect that containers would have taken off even without isolation. I think the important innovation of Docker was the image. It let people deploy consistent version of their software or download outside software. All of the hassle of installing things was in the Dockerfile, and it was run in containers so more reliable.
    - 1y ago
aPoCoMiLogin 1y ago
it happened because the story of dependencies (system & application) was terrible. the ability to run the app on different distribution/kernel/compiler/etc was hard. there were different solutions like vagrant, but they were heavy and the DX wasn't there
jcelerier 1y ago
For me the main reason to use containers is "one-line install any linux distro userspace". So much simpler than installing a dozen VirtualBox boxes to test $APP on various versions of ubuntu, debian, nixos, arch, fedora, suse, centos etc.
- kccqzy 1y ago
  Yeah nowadays we have the distrobox(1) command. Super useful. But certainly that's not why containers happened.
guigar 1y ago
I love this sentence about DevOps "Somehow it seems easier for people to relate to technology than culture, and the technology started working against the culture."
- draw_down 1y ago
  [dead]
chatmasta 1y ago
My headcanon is that Docker exists because Python packaging and dependency management was so bad that dotCloud had no choice but to invent some porcelain on top of Linux containers, just to provide a pleasant experience for deploying Python apps.
- frumplestlatz 1y ago
  Pretty much this; systems with coherent isolated dependency management, like Java, never required OS-level container solutions. They did have what you could call userspace container management via application servers, though.
  - drowsspa 1y ago
    NodeJS, Ruby, etc also have this problem, as does Go with CGO. So the problem is the binary dependencies with C/C++ code and make, configure, autotools, etc... The whole C/C++ compilation story is such a mess that almost 5 decades ago inventing containers was pretty much the only sane way of tackling it. Java at least uses binary dependencies very rarely, and they usually have the decency of bundling the compiled dependencies... But it seems Java and Go just saw the writing on the wall and mostly just reimplement everything. I did have problems with the Snappy compression in the Kafka libraries, though, for instance .
    - skydhash 1y ago
      The issue is with cross platform package management without proper hooks for the platform themselves. That may be ok if the library is pure, but as soon as you have bindings to another ecosystem (C/C++ in most cases), then it should be user/configurable instead of the provider doing the configuration with post installs scripts and other hacky stuff. If you look at most projects in the C world, they only provide the list of dependencies and some build config Makefile/Meson/Cmake/... But the latter is more of a sample and if your platform is not common or differs from the developer, you have the option to modify it (which is what most distros and port systems do). But good luck doing that with the sprawling tree of modern packages managers. Where there's multiple copies of the same libraries inside the same project just because.
spullara 1y ago
Because dependencies on Unix are terrible for some languages that assume things are installed globally.
Hizonner 1y ago
Original sin.
BandButcher 1y ago
"The compute we are wasting is at least 10x cheaper, but we have automation to waste it at scale now." So much this. keep it simple, stupid (muah)
figassis 1y ago
> I was always surprised someone didn't invent a tool for ftping to your container and updating the PHP We thought of it, and were thankful that it was not obvious to our bosses, because lord forbid they would make it standard process and we would be right back where we started, with long lived images and filesystem changes, and hacks, and managing containers like pets.
tacker2000 1y ago
The author suggests that Docker doesnt help development and that devs just spin up databases, but I have to disagree with that and Im pretty sure i am not the only one. All my projects (primarily web apps) are using docker compose which configures multiple containers (php/python/node runtime, nginx server, database, scheduler, etc) and run as a dev environment on my machine. The source code is mounted as a volume. This same compose file is then also used for the deployment to the production server (with minor changes that remove debug settings for example). This approach has worked well for me as a solo dev creating web apps for my clients. It has also enabled extreme flexibility in the stacks that I use, I can switch dev environments easily and quickly.
- boomskats 1y ago
  I agree with you 100%, though arguably what you could be describing is how docker changed your deployment workflow, not your development workflow (although with devcontainers that line is blurry, as you say). I guess it's worth keeping in mind that Justin only quit Docker a few months ago, and his long tenure as CTO there will have (obviously) informed the majority of the opinions in the article. I think the deployment over development spin and some of the other takes there more closely reflect the conversations he had with large corp paying customers at the exec level than the workflows of solo devs that switch dev environments much more frequently than most etc.
  - tacker2000 1y ago
    It goes hand in hand, it changed both the dev and he deploy workflow. Before Docker I was using Xampp and FTP’ing source code to the prod server.
    - man8alexd 1y ago
      Alternatives to FTP'ing, as deploying from a source code repository and tools like capistrano, heroku, chef deploy resource, puppet vcsrepo existed long before Docker.
      - 1y ago
lisbbb 1y ago
I loved the assertion that AI ate up all the budget and that K8s is now "boring" technology. That's fine because it was getting pretty annoying with all the clone competitors for practically everything that were popping up every month! Do you use K8s? No! That's old! I use Thrumba! It's just a clone of K8s by some startup because people figured out that the easiest way to make gobs of money is/was to build platform products and then get people to use them.
IshKebab 1y ago
Because Linux devs generally suck at making portable packages that are easy to install.
bane 1y ago
Containers happened because nobody can be bothered to build an entire application into a single distributable executable anymore - heck even the tooling barely exists anymore. But instead of solving problems like dependency management and linking, today's engineers simply build piles of abstraction into the problem space until the thing you want to do more than anything (i.e. execute an application) becomes a single call. Of course you now need to build and maintain those abstract towers, so more jobs for everybody!
- gorfian_robot 1y ago
  this is what happens when hw is too cheap
  - zbentley 1y ago
    You sure? Which hardware? Put another way: stuff like Electron makes a pretty good case for the "cheap hardware leads to shitty software quality/distribution mechanisms" claim. But does Docker? Containers aren't generally any more expensive in hardware other than disk-space to run than any other app. And disk space was always (at least since the advent of the discrete HDD) one of the cheapest parts of a computer to scale up.
    - wmf 1y ago
      If you go back to the Sun days, you literally could not afford enough servers to run one app per server so instead you'd hire sysadmins to figure out how to run Sendmail and Oracle and whatever on one server without conflicting. Then x86/Linux 1Us came out and people started just running one app per server ("server sprawl") which was easy because there was nothing to conflict. This later became VM sprawl and containers were an optimization on that.
      - zbentley 1y ago
        I'm not getting it, sorry. We had to have multiple apps per server before, and now we have containers which offer a convenient way to have multiple apps per server? That seems like the same thing. Could you explain more re: what you meant?
wucke13 1y ago
My take: containers forced devepopers to declare various aspects of the application in a standardized, opinioated way: - Persistant state? Must declare a volume. - IO with external services? Must declare the ports (and maybe addresses). - Configurable parameters? Must declare some env variables. - Trasitive dependecies? Must declare them, but using a mechanism of your choosing (e.g. via the package manager of your base image distro). Separation of state (as in persistency) and application (as in binaries, assets) makes updates easy. Backups also. Having most all IO visible and explicit simplifies operation and integration. And a single, (too?!?) simple config mechanism increases reusability, by enabling e.g. lightweight tailoring of generic application service containers (such as mariadb). Together this bunch of forced, yet leaky abstractions is just good enough to foster immense reuse & composability on to a plethora of applications, all while allowing to treat them almost entirely like blackboxes. IMHO that is why OCI containers became this big, compared to other virtualization and (application-) cuntainer technologies.
- ranger207 1y ago
  Yep, this is why. Containers are a way to package more of the OS environment than you can do otherwise
- lotyrin 1y ago
  Yeah. I remember when "okay, cool, how do we deploy, run, maintain the thing we (our leadership) just bought from you" was several weeks of meetings with the vendor and discussing awful "human-readable" documentation (if it existed). Those meetings still happen sometimes, but "cloud" *aaS and containers really put a dent in that sort of thing.
0xbadcafebee 1y ago
Why it happened is not nearly as important as what it unveiled: that versioned immutable systems are the most powerful system design concept in history. Most people have not yet grasped what an insanely powerful concept it is. At some point in the future, maybe 50-100 years from now, someone will look back and say "holy shit; deploying web apps are where this concept came from?" I hope in my lifetime that people get it and start applying it to other system designs/components.
- hackshack 1y ago
  radmind had this philosophy. https://radmind.org https://radmind.org It let you create diffs of a filesystem, and layer them with configurations, similar to containers. Useful for managing computer labs at the time.
  - 0xbadcafebee 1y ago
    Well disk (or tape) images are as old as 'dd' and 'tar', that's not the revolutionary part. If the disk is writeable the state is still constantly mutating, so you're fighting a war of attrition (it's configuration management at that point, which is terrible). But a read-only disk that doesn't accrue changes, and only needs a reboot to fix, that's the revolutionary part. Anybody who ran thin-terminals can tell you how reliable and easier to manage those are than a full-blown OS. At some point in the future people are going to realize that every system should work that way.
gorfian_robot 1y ago
in some situations, the cloud is a no-go and the lead time to install new hardware is quite lengthy. in those case vm's or containers allow for rapid change at the software/OS level while the meat space still moves like a glacier. sounds minor, but it is a Big Deal for some
nkozyra 1y ago
> was always surprised someone didn't invent a tool for ftping to your container and updating the PHP. No FTP needed, you can just mount the application directory.
daxfohl 1y ago
> Application composition from open source components became the dominant way of constructing applications over the last decade. I'm just as interested in why this ^ happened. I imagine it's pretty unique to software? I don't hear of car companies publishing component designs free for competitors to use, or pharmaceuticals freely waiving the IP in their patents or processes. Certainly not as "the dominant way of" doing business. I wonder if LLM coding assistants had come about earlier, whether this would have been as prevalent. Companies might have been more inclined to create more of their own tooling from scratch since LLMs make it cheap (in theory). Individuals might have been less inclined to work on open source as hobbies because LLMs make it less personal. Companies might be less inclined to adopt open-source LLM-managed libraries because it's too chaotic.
- travisjungroth 1y ago
  I think open source software took off because it’s more standalone than the other things you listed and this makes the rewards much higher. If I write some code, it needs a computer and environment to run. If I’m writing for what’s popular, that’s pretty much a given. In short, for code the design is the product. If I design a pharmaceutical, someone still has to make it. Same for car parts. This effort is actually greater than the effort of design. If you include regulation, it’s way higher. So, this great feedback loop of creation-impact-collaboration never forms. The loop would be too big and involve too much other stuff. The closest thing isn’t actually manufacturing, it’s more like writing and music. People have been reusing each other’s stuff forever in those spaces.
somat 1y ago
containers happened because the original execution isolation environment(the process) was considered a lost cause, Processes shared too much with each other so additional isolation primitives had to be added, but they had to be sort of tacked on to the side because more important than security or correctness is backwards compatibility. so now containers are considered a different thing than processes when really they are process with these additional isolation primitives enabled.
- ghaff 1y ago
  In the early 2000s (yes, long after the original jails), containers were pitched as an alternative to VMware's VMs. They lost out for a variety of reasons--but mostly because as purely a different (and somewhat lighter-weight) encapsulation technique they weren't that interesting.
  - deleted 1y ago
    [deleted]
forrestthewoods 1y ago
Containers were invented because Linux is incapable of reliably running software programs. The global pool of shared libraries is an abject failure and containers are a heavy handed workaround.
crummy 1y ago
> Docker also made Go credible as a programming language, Can someone explain why Docker was beneficial for Go?
- jopsen 1y ago
  Because docker was written in golang. And before docker, not many large applications were.
  - freetonik 1y ago
    Ironically, also because Go is one of few popular languages for web applications that can produce a single executable binary and does not require a container to deploy with ease.
  - CamouflagedKiwi 1y ago
    I think there's a pretty big citation needed on that part of the article. I'm not clear that Docker contributed to that anywhere near as much as a general increase of momentum around Go as it became better known in the industry.
- nitwit005 1y ago
  I've never heard of anyone basing their choice to use Go off of the fact that Docker used it. Certainly, the fact that Google was pushing Go mattered far more. I think you should see this article as someone who's a huge fan of Docker due to working there for years.
coppsilgold 1y ago
When you launch a container (either through docker you manually through namespaces) you are effectively representing yourself to the kernel as a separate thing. This allows you to construct a completely separate environment when interacting with the kernel where none of your concerns are going to leak out and nothing you don't care for is going to leak in. When people say that static executables would solve the problem they are wrong, a static executable just means that you can eschew constructing a separate file-system inside your container - and you will probably need to populate some locations anyway. Properly configured containers are actually supposed to be secure sandboxes, such that any violation is a kernel exploit. However the Linux kernel attack surface is very large so no one serious who offers multi-tenant hosting can afford to rely on containers for isolation. They have to assume that a container escape 0day can be sourced. It may be more accurate to say that a general kernel 0day can be sourced since the entire kernel surface area is open for anyone to poke. seccomp can mitigate the surface area but also narrow down the usefulness.
- 112233 1y ago
  not .. really. Linux kernel has no concept of a container, you have to be super careful to avoid "mixing" host stuff in. I'm yet to see an case where "leaking in" would be prevented by default. Docker "leaks in" as much as you want. Containers also do not nest gracefully (due to, e.g., uids), so cannot be used as a software component. It's mostly a linux system admin thing right now.
  - coppsilgold 1y ago
    The Linux kernel provides the namespaces... Docker has made some strange decisions for default behavior but if you take a more hands on approach such as with bubblewrap/bwrap nothing will leak in.
    - 112233 1y ago
      How would you do it? I'm quite interested! How can you hide container processes in host procfs using bwrap? And make sure no mounts stay mounted in the host? The most "nothing leaks in" runtime I've seen is gVisor (before going VM). Attaining that with bwrap would be nice, but I'm sceptical.
lloeki 1y ago
> There was one key innovation, which was Docker Hub Vagrant has had that for VMs long before. Dockerfiles and Docker Hub are directly inspired from Vagrantfiles and the Vagrant Box library
01HNNWZ0MV43FF 1y ago
I just wanted to have two instances of two versions of Postgres installed, and have their data directories documented in a YAML file, and know that they aren't gonna fuck up anything else on my system if someone gets RCE or something odd
ksk23 1y ago
I love boring.
w10-1 1y ago
This and the comments may miss the forest for the trees. Enterprise software vendors sold libraries and then "application servers", essentially promising infrastructure (typically tied to databases). Enterprise software developers -- Google in particular -- got tired of depending on others' licensed infrastructure. This birthed Spring and Docker, splitting the market. (Fun aside: when is a container a vm? When it runs via Apple containerization.)
cyberax 1y ago
The article is just wrong. Before Docker, there was OpenVZ and Virtuozzo. They were used to provide cheaper "dedicatd machine" hosting back around 2005. Then the technology from OpenVZ slowly made its way into the mainline Linux, in the form of cgroups and namespaces. LWN called it a "container puzzle", with tens of moving pieces. And it was largely finished by early 2010-s. I built my own container system in 2012 that used cgroups to oversubscribe the RAM, with simple chroot-based file namespaces for isolation. We even used XFS projects (raise your hand if you know what this is!) for the disk quota management. I remember that I had to use systemtap to patch the kernel to be able to find out which process died as a result of the OOM killer, there were no standard ways to do that. We sold it as a part of our biotech startup to Illumina. Then we sold it again to Amazon as a part of another startup :) The genius of Docker was the layered overlayfs-based image building. This one simple innovation made it possible to build images in a constructive way, without having to waste half an hour for each minor change. I was floored with its simplicity and power when I first saw it.
- man8alexd 1y ago
  OpenVZ didn't use cgroups but its own resource limits, which sucked.
  - cyberax 1y ago
    Well, yes. "Beancounters" and it also was not nestable. That's why it took quite a bit of time to get it all upstreamed.
    - man8alexd 1y ago
      Unless my memory fails me, OpenVZ was never upstreamed.
Joker_vD 1y ago
> What people do with Docker is spin up a database or another service to develop or test against. Yep. Being able to run docker run --rm --publish=127.0.0.1:27017:27017 'mongo:3.6.8' or docker run --rm --publish=127.0.0.1:27017:27017 'mongo:5.0' and then get rid of it with simple Ctrl-C is a godsend.
- DanielHB 1y ago
  I remember installing Oracle in my desktop computer to be able to do my programming classes... If the installation crashed (which was quite common, happened to me once) it was easier to just format the computer completely and start again. Effed up the database? Probably easier to format everything and install from scratch
dilyevsky 1y ago
Linux CGroups specifically were started at Google because their cluster management system Borg (or maybe it was still Babysitter at the time) needed a way to do resource tracking and admission control. Here's a comment by one of original devs: https://news.ycombinator.com/item?id=25017753 https://news.ycombinator.com/item?id=25017753
- deviation 1y ago
  This is also somewhat highlighted in Google's paper "Borg, Omega, and Kubernetes" which they published in 2016. https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/44843.pdf https://static.googleusercontent.com/media/research.google.c...
boomskats 1y ago
Fwiw the actual video that he links to is well worth a watch. Featuring one of the most Justin intros ever.
alexey-salmin 1y ago
The article seems to assume that containers appeared to solve the software distribution problem and then somehow got repurposed into virtualization, isolation and management of production services. I think this view is very far from truth. The virtualization/isolation aspect came first, the SWSoft Virtuozzo was doing that quite well in early 2000s. They even had [some] IO isolation which I think took around a decade to support elsewhere. Then gradually pieces of Virtuozzo/OpenVZ reached the mainline in a form of cgroups/LXC and the whole thing slowly brewed for a while until the Docker added the two missing pieces: the fast image rebuilds and the out-of-the-box user experience. Docker of course was the revolution, but by then sufficiently advanced companies have been already using containers for isolation for a full decade.
- bayindirh 1y ago
  I believe BSD Jails created a "Why don't we have this under Linux" feeling as well.
- rmoriz 1y ago
  To my knowledge the very first appearance was the "ctx" kernel patches project by linux-vserver.org by Herbert Pötzl et al. https://web.archive.org/web/20250307173707/http://linux-vserver.org/Welcome_to_Linux-VServer.org https://web.archive.org/web/20250307173707/http://linux-vser... (sadly, the domain seems to be expired just recently) I used it heavily back in the days, long before kernel namespaces went upstream and docker became a thing.
- DanielHB 1y ago
  I remember hearing about all the support from Intel for "0-cost" virtualization at the hardware level way before I heard about containers. From what I remember it was mostly to speed up virtual machines (VMWare stuff). It was a massive market differentiator for Intel in the server space. I vaguely remember having to turn on some features in VirtualBox at the time to speed up my VMs, it was a massive uplift in performance if you had a CPU that supported it.
kotaKat 1y ago
Containers happened because nobody knew what the hell they were doing and still have no clue what the hell they are doing. Software by the deranged for the deranged.
- goodpoint 1y ago
  ...and they don't care about security.
0dayz 1y ago
What blew my mind and convinced me to only use immutable distros is the immutability of it. For instance I could create my own login screen for an web service without having to worry about the package manager overriding my code, because I inject it into the container, which is already updated. I can also forcefully reroute much easier ports or network connections the way I want it.
lazylizard 1y ago
how did solaris zones get left out of the story?
- lproven 1y ago
  George Santayana and Henry Spencer can tell you. Santayana in 1905: "Those who cannot remember the past are condemned to repeat it." Spencer in 1987: "Those who do not understand Unix are condemned to reinvent it, poorly."
bboreham 1y ago
The tool that connects into your container and updates the files was called "Tilt".
nahuel0x 1y ago
Because containers were lighter than VM's, made reproducible and modularizable how to build the VM contents, and "solved" Unix/POSIX process isolation / library management design flaws.
yohbho 1y ago
> Docker was easy to adopt as it did not change very much about how you used software. What?! Docker images should seperate the program from the data. Something casual image maintainers fail to adhere to. I find docker extreme hard to use for persistent state that is not a remote DB. It is even less secure than VMs (which seem to be like swiss chess if you try to break out). (Security-wise: rkt failed five years ago, BSD jails could do it better?) Also, Docker was supposed to solve performance issues of VMs, debloating them. Instead, we got hypervisors or full VMs as hosts for Docker images, just to get the image as a de facto meta package manager to ship software. So for me, it can be convenient, but I have no idea what I am doing when adding make file like statements to a yml to run full containers just to run e.g. a web server with some dependecies. But the workflow for that was novell to me, who previously fought with ruby and python module installations on host system.
lproven 1y ago
Money. Containers were around for a decade or more on FreeBSD and Solaris. They let you divvy up expensive big Unix iron. Same as VMs were around on mainframes from the late 1960s and expensive Unix RISC servers from the late 1980s. Linux didn't need it because it was cheap. So Linux replaced that older more expensive stuff, on cheap COTS hardware: x86. Once everything was commoditised and cost-cut, suddenly, efficiency started to matter, so VMware thrived and was copied and VMs were everywhere. Then the low usage and inefficiency of resource sharing of VMs made them look expensive, so they started to get displaced by the cheaper easier tech of containers, making "it works on my machine, so let's ship my machine" scale to production.
- pjmlp 1y ago
  Even earlier, I was introduced to the concept of containers in HP-UX Vault back in 1999, before FreeBSD and Solaris got them. Unfortunely HPe has removed most of HP-UX documentation out of Internet so it is hard to point it out. However there are still some digital traces, HP-UX 10.24 release, > This is a Virtual Vault release of HP-UX, providing enhanced security features. Virtual Vault is a compartmentalised operating system in which each file is assigned a compartment and processes only have access to files in the appropriate compartment and unlike most other UNIX systems the superuser (or root) does not have complete access to the system without following correct procedures. https://en.wikipedia.org/wiki/HP-UX https://en.wikipedia.org/wiki/HP-UX A forum discussion, https://community.hpe.com/t5/operating-system-hp-ux/hp-virtual-vault/td-p/3865588 https://community.hpe.com/t5/operating-system-hp-ux/hp-virtu... Now it would be great to get back those HP-UX Vault PDFs.
  - lproven 1y ago
    Fair enough. I never worked on HP/UX. AIX, Solaris, Xenix, and VMS, OS/400, and RSTS, but never HP. I wrote about containers being the next big thing in 2011: https://www.theregister.com/Print/2011/07/18/brief_history_of_virtualisation_part_3/ https://www.theregister.com/Print/2011/07/18/brief_history_o... I credit FreeBSD and Solaris and AIX, but I think I didn't know about HP/UX.
    - 1y ago
rdsubhas 1y ago
I used cgroups, lxc, chroots, self-extracting executables. I built rugged, portable applications for UNICEF laptops and camps before docker was a thing. And I think this whole point about "virtualization", "security", making the most use of hardware, reducing costs, and so on, while true, it's an "Enterprise pitch" targeted at heads of tech and security. Nice side effects, but I couldn't care less. There are real, fundamental benefits to containers for a solo developer running a solo app on a solo server. Why? My application needs 2 or 3 other folders to write or read files into, maybe 2 or 3 other runtime executables (jvm, node, convert, think of the dozens of OSS CLI tools, not compile-time libraries), maybe apt-get install or download a few other dependencies. Now I, as a indie developer, can "mkdir" a few files from a shell script. But that "mkdir" will work the first time. It will fail the second time saying "directory already exists". I can "apt-get install" a few things, but upgrading and versioning is a different story altogether. It's a matter of time before I realize I need atleast some barebones ansible or state management. I can tell you many times how I've reinvented "smallish" ansible in shell scripts before docker. Now if I'm in an enterprise, I need to communicate this entire State of my app – to the sysadmin teams. Forget security and virtualization and all that. I need to explain every single part of the state, versions of java and tomcat, the directories, and all those are moving targets. Containers reduce state management. A LOT. I can always "mkdir". I can always "apt-get install". It's an ephemeral image. I don't need to write half-broken shell scripts or use ansible or create mini-shell-ansible. If you use a Dockerfile with docker-compose, you've solved 95% of state management. The only 5% left is to docker-compose the right source. Skip the enterprisey parts. A normal field engineer or solo developer, like me, who's deploying a service on the field, even on my raspberry pi, would still use containers. It boils down to one word: "State management" which most people completely underestimate as "scripting". Containers grant a large control on state management to me, the developer, and simplify it by making it ephemeral. That's a big thing.
- jiggunjer 1y ago
  That's two words. How about "deterministic".
  - abound 1y ago
superkuh 1y ago
Pretty simple. Modern development cycles for the libs that software depends on now happen significantly faster that OS/Distros update. This is future shock. The symptoms of this disease express as containers. Do your part. Don't code on the bleeding edge. Keep things stable for users.
marsven_422 1y ago
[dead]
stego-tech 1y ago
A really good post that captures the IT perspective of infrastructure change and modernization over my career. They also hit the nail on the head that one of today's problems is itself automation, where we've built tools that let us create sprawling estates with no accountability for cost other than an opaque monthly bill, though I'd also chip in that said automation is often restricted to whoever your chief cloud provider is (AWS, Azure, etc). Where I find myself advocating today is very much a "rational check" on infrastructure, and curtailing accordingly. We have the tooling to ensure high availability, but does everything need to be HA? Do our SLAs for enterprise tooling really need five-nines of availability, or can we knock some applications down to a limited schedule? Does dev/test need to be live 24/7, or can we power it off when not in use? Why are we only focusing on availability and not scalability? The list goes on, but they're also not popular in enterprises with entrenched politics, which admittedly is where I find myself struggling against the current. If my social chops were better, I suspect I'd thrive in consultancy doing just that. All that being said, I do like containers that are done right (properly documented, secure-by-default, ready for scaling), and I continue driving more applications towards containerization in the enterprise where possible. They're the right solution for ~60-80% of enterprise use cases, with the difficulty being getting vendors on board with the idea that their software won't have a dedicated VM or hardware anymore (which everyone fiercely resists, because container-based licensing can be a PITA to them). For the rest, VMs are more than fine, and we have a growing number of ways for both to exist peacefully in the same environment. As this area of technology matures (along with "backporting" from hyperscalers to private cloud again) further, I'm really looking forward to managing global estates in smaller teams for bigger firms - things that VMs, Containers, and Infrastructure-as-Code allow.
sharts 1y ago
Why does every discussion about containers always end up revolving around linux and cgroups? Didn’t we have Zones and even Jails way back?
majke 1y ago
Most of the responses here are knee-jerk. I liked the article. It's close to my adverntures with containers. I think the invention of docker is indeed mostly packaging: Dockerfile (functional) is pretty neat, docker hub (addressing a container) is awesome, and the ENTRYPOINT in Dockerfile is great, it distinguishes Docker from .deb. But indeed, beyond Dockerfile things are bleak. Docker compose never rose to my expectations. To get serious things you need load blancer, storage, addressing, and these are beyond traditional containers scope.
Saris 1y ago
I have no idea how close I am, but Linux seems to have no way of having multiple versions of a dependency at once like Windows does. So I suspect thats the main reason.

208 points by todsacerdoti 1y ago

https://www.youtube.com/watch?v=eMU2mZgo99c https://www.youtube.com/watch?v=eMU2mZgo99c

all2 1y ago

Likely because Plan9's 'everything-is-a-filesystem' failed.

jauntywundrkind 1y ago
Never grew popular, perhaps. But I'm not sure how it failed, and not sure how many of the Venm Diagrams of concerns plan9 really has with containers. Yes there was an idea of creating bespoke filesystems for apps, custom mount structures that plan9 had. That containers also did something semi-parallel to. But container images as read only overlays (with a final rw top overlay) feel like a very narrow craft. Plan9 had a lot more to it (everything as a file), and containers have a lot more to them (process, user, net namespaces, container images to pre-assembled layers). I can see some shared territory but these concerns feel mostly orthogonal. I could easily imagine a plan9 like entity arising amid the containerized world: these aren't really in tension with each other. There's also a decade and a half+ gap between Plan9's hayday and the rise of containers.
walkabout 1y ago
The standard answer is, "because inventing and implementing them was easier than fixing Python packaging."
- aaroninsf 1y ago
  Ngl this is why I started using them
- LexiMax 1y ago
  I think "fixing distro packaging" is more apropos. In a past life, I remember having to juggle third-party repositories in order to get very specific versions of various services, which resulted in more than few instances of hair-pull-inducing untangling of dependency weirdness. This might be controversial, but I personally think that distro repos being the assumed first resort of software distribution on Linux has done untold amounts of damage to the software ecosystem on Linux. Containers, alongside Flatpak and Steam, are thankfully undoing the damage.
  - bombcar 1y ago
    I wonder if it can be traced back to something RedHat did somewhere, because it may have all begun once you COULDN'T be absolutely certain that anything even remotely "enterprise" was running on a RedHat.
    - LexiMax 1y ago
      I think it's a natural outgrowth of what Linux is. Linux is just a kernel - you need to ship your own userland with it. Therefore, early distros had to assemble an entire OS around this newfangled kernel from bits and pieces, and those bits and pieces needed a way to be installed and removed at will. Eventually this installation mechanism gets scope creep and and suddenly things like FreeCiv and XBill are distributed using the same underlying system that bash and cron use. This system of distro packaging might be good as a selling point for a distro - so people can brag about their distro comes with 10,000 packages or whatever. That said, I can think of no other operating system out there where the happiest path of releasing software is to simply release a tarball of the source, hope a distro maintainer packages it for you, hope they do it properly, and hope that nobody runs into a bug due to a newer or older version of a dependency you didn't test against.
      - skydhash 1y ago
        Yours is a philosophy I encounter more and more. Where there should be that unified platform, ideally fast moving, where software is only tested against $latest. Stability is a thing of the past. The important thing is more feature. Instead of designing a solution and perfecting it overtime, it's endless tweaking where there's a new redesign every years. And you're supposed to use the exact computer as the Dev to get their code to work.
    - ghaff 1y ago
      Red Hat was actually doing something more directly based on a variety of existing Linux projects than Docker but switched to OCI/Docker when that came about--rather than jumping on the CloudFoundry bandwagon. (Which many argues was obviously the future for container orchestration.) Kubernetes was also not the obvious winner in its time with Mesos in particular seeming like a possible alternative when it wasn't clear if orchestration and resource management weren't possibly different product categories. I was at Red Hat at the time and my impression was they did a pretty good job of jumping onto where the community momentum at the time was--while doubtless influencing that momentum at the time.

jmclnx 1y ago

FreeBSD jails years ago based upon a user request. >hosting provider's ... desire to establish a clean, clear-cut separation between their own services and those of their customers https://en.wikipedia.org/wiki/FreeBSD_jail https://en.wikipedia.org/wiki/FreeBSD_jail My guess Linux started getting requests rom various orgs for a while, so in true Linux fashion, we got a a few different container type methods years later. I still think Jails are the best of the bunch, but they can be a bit hard to setup. Once setup, Jails works great. So here we are :)

IlikeKitties 1y ago

I mean, containers do lend themselves to cargo culting by their very nature.

NewJazz 1y ago
https://youtube.com/watch?v=TwxetMVdYTc&t=30s https://youtube.com/watch?v=TwxetMVdYTc&t=30s

blu3h4t 1y ago

You can laugh or not but its because they never finished gnu/hurd :D

hedgehog 1y ago

Containers happened because running an ad network and search engine means serving a lot of traffic for as little cost as possible, and part of keeping the cost down is bin packing workloads onto homogeneous hardware as efficiently as possible. https://en.wikipedia.org/wiki/Cgroups https://en.wikipedia.org/wiki/Cgroups (arguably FreeBSD jails and various mainframe operating systems preceded Linux containers but not by that name)

cbdumas 1y ago
What does the 'ad network and search engine' have to do with it? Wouldn't any organization who serves lots of traffic have the same cost cutting goals you mentioned?
- wmf 1y ago
  It's an oblique way to say that Linux cgroups and namespaces were developed by Google.
  - hedgehog 1y ago
    Yes, to expand: Both search and ads mean serving immense amounts of traffic and users while earning tiny amounts of revenue per unit of each. The dominant mid-90s model of buying racks of Sun and NetApp gear, writing big checks to Oracle, etc, would have been too expensive for Google. Instead they made a big investment in Linux running on large quantities of commodity x86 PC hardware, and building software on top of that to get the most out of it. That means things like combining workloads with different profiles onto the same servers, and cgroups kind of falls out of that. Other companies like Yahoo, Whatsapp, Netflix also followed interesting patterns of using strong understanding of how to be efficient on cheap hardware. Notably those three all were FreeBSD users at least in their early days.
    - dmoy 1y ago
      Yup and just to add timelines - Google Borg and containerization was what... 2003-2005? Docker was 2011-2013?

rshnotsecure 1y ago

Fascinating documentary on Kubernetes for those who have 50 minutes. Gives more background to the "Container Wars". The filmmakers also have documentaries on the history of Python, Argo, etc. Some highlights: - How far behind Kubernetes was at the time of launch. Docker Swarm was significantly more simple to use, and Apache Mesos scheduler could already handle 10,000 nodes (and was being used by Netflix). - RedHat's early contributions were key, despite having the semi-competing project of OpenShift. - The decision to Open Source K8S came down to one meeting brief meeting at Google. Many of the senior engineers attended remotely from Seattle, not bothering to fly out because they thought their request to go OS was going to get shutdown. - Brief part at the end where Kelsey Hightower talks about what he thinks might come after Kubernetes. He mentions, and I thought this was very interesting ... Serverless making a return. It really seemed like Serverless would be "the thing" in 2016-2017 but containers were too powerful. Maybe now with KNative or some future fusing of Container Orchestration + K8S? [1] - https://youtu.be/BE77h7dmoQU https://youtu.be/BE77h7dmoQU

btreecat 1y ago
I feel that's going to be more interesting than this video. The speaker is very unpracticed.

alphazard 1y ago

Containers (meaning Docker) happened because CGroups and namespaces were arcane and required lots of specialized knowledge to create what most of us can intuitively understand as a "sandbox". Cgroups and namespaces were added to Linux in an attempt to add security to a design (UNIX) which has a fundamentally poor approach to security (shared global namespace, users, etc.). It's really not going all that well, and I hope something like SEL4 can replace Linux for cloud server workloads eventually. Most applications use almost none of the Linux kernel's features. We could have very secure, high performance web servers, which get capabilities to the network stack as initial arguments, and don't have access to anything more. Drivers for virtual devices are simple, we don't need Linux's vast driver support for cloud VMs. We essentially need a virtual ethernet device driver for SEL4, a network stack that runs on SEL4, and a simple init process that loads the network stack with capabilities for the network device, and loads the application with a capability to the network stack. Make building an image for that as easy as compiling a binary, and you could eliminate maybe 10s of millions of lines of complexity from the deployment of most server applications. No Linux, no docker. Because SEL4 is actually well designed, you can run a sub kernel as a process on SEL4 relatively easily. Tada, now you can get rid of K8s too.

tptacek 1y ago
This makes sense if you look at containers as simply a means to an end of setting up a sandbox, but not really much sense at all if you think of containers as a way to make it easy to get an arbitrary application up and running on an arbitrary server without altering host system dependencies.
- ianburrell 1y ago
  I suspect that containers would have taken off even without isolation. I think the important innovation of Docker was the image. It let people deploy consistent version of their software or download outside software. All of the hassle of installing things was in the Dockerfile, and it was run in containers so more reliable.
  - 1y ago

aPoCoMiLogin 1y ago

it happened because the story of dependencies (system & application) was terrible. the ability to run the app on different distribution/kernel/compiler/etc was hard. there were different solutions like vagrant, but they were heavy and the DX wasn't there

jcelerier 1y ago

For me the main reason to use containers is "one-line install any linux distro userspace". So much simpler than installing a dozen VirtualBox boxes to test $APP on various versions of ubuntu, debian, nixos, arch, fedora, suse, centos etc.

kccqzy 1y ago
Yeah nowadays we have the distrobox(1) command. Super useful. But certainly that's not why containers happened.

guigar 1y ago

I love this sentence about DevOps "Somehow it seems easier for people to relate to technology than culture, and the technology started working against the culture."

draw_down 1y ago
[dead]

chatmasta 1y ago

My headcanon is that Docker exists because Python packaging and dependency management was so bad that dotCloud had no choice but to invent some porcelain on top of Linux containers, just to provide a pleasant experience for deploying Python apps.

frumplestlatz 1y ago
Pretty much this; systems with coherent isolated dependency management, like Java, never required OS-level container solutions. They did have what you could call userspace container management via application servers, though.
- drowsspa 1y ago
  NodeJS, Ruby, etc also have this problem, as does Go with CGO. So the problem is the binary dependencies with C/C++ code and make, configure, autotools, etc... The whole C/C++ compilation story is such a mess that almost 5 decades ago inventing containers was pretty much the only sane way of tackling it. Java at least uses binary dependencies very rarely, and they usually have the decency of bundling the compiled dependencies... But it seems Java and Go just saw the writing on the wall and mostly just reimplement everything. I did have problems with the Snappy compression in the Kafka libraries, though, for instance .
  - skydhash 1y ago
    The issue is with cross platform package management without proper hooks for the platform themselves. That may be ok if the library is pure, but as soon as you have bindings to another ecosystem (C/C++ in most cases), then it should be user/configurable instead of the provider doing the configuration with post installs scripts and other hacky stuff. If you look at most projects in the C world, they only provide the list of dependencies and some build config Makefile/Meson/Cmake/... But the latter is more of a sample and if your platform is not common or differs from the developer, you have the option to modify it (which is what most distros and port systems do). But good luck doing that with the sprawling tree of modern packages managers. Where there's multiple copies of the same libraries inside the same project just because.

spullara 1y ago

Because dependencies on Unix are terrible for some languages that assume things are installed globally.

Hizonner 1y ago

Original sin.

BandButcher 1y ago

"The compute we are wasting is at least 10x cheaper, but we have automation to waste it at scale now." So much this. keep it simple, stupid (muah)

figassis 1y ago

> I was always surprised someone didn't invent a tool for ftping to your container and updating the PHP We thought of it, and were thankful that it was not obvious to our bosses, because lord forbid they would make it standard process and we would be right back where we started, with long lived images and filesystem changes, and hacks, and managing containers like pets.

tacker2000 1y ago

The author suggests that Docker doesnt help development and that devs just spin up databases, but I have to disagree with that and Im pretty sure i am not the only one. All my projects (primarily web apps) are using docker compose which configures multiple containers (php/python/node runtime, nginx server, database, scheduler, etc) and run as a dev environment on my machine. The source code is mounted as a volume. This same compose file is then also used for the deployment to the production server (with minor changes that remove debug settings for example). This approach has worked well for me as a solo dev creating web apps for my clients. It has also enabled extreme flexibility in the stacks that I use, I can switch dev environments easily and quickly.

boomskats 1y ago
I agree with you 100%, though arguably what you could be describing is how docker changed your deployment workflow, not your development workflow (although with devcontainers that line is blurry, as you say). I guess it's worth keeping in mind that Justin only quit Docker a few months ago, and his long tenure as CTO there will have (obviously) informed the majority of the opinions in the article. I think the deployment over development spin and some of the other takes there more closely reflect the conversations he had with large corp paying customers at the exec level than the workflows of solo devs that switch dev environments much more frequently than most etc.
- tacker2000 1y ago
  It goes hand in hand, it changed both the dev and he deploy workflow. Before Docker I was using Xampp and FTP’ing source code to the prod server.
  - man8alexd 1y ago
    Alternatives to FTP'ing, as deploying from a source code repository and tools like capistrano, heroku, chef deploy resource, puppet vcsrepo existed long before Docker.
    - 1y ago

lisbbb 1y ago

I loved the assertion that AI ate up all the budget and that K8s is now "boring" technology. That's fine because it was getting pretty annoying with all the clone competitors for practically everything that were popping up every month! Do you use K8s? No! That's old! I use Thrumba! It's just a clone of K8s by some startup because people figured out that the easiest way to make gobs of money is/was to build platform products and then get people to use them.

IshKebab 1y ago

Because Linux devs generally suck at making portable packages that are easy to install.

bane 1y ago

Containers happened because nobody can be bothered to build an entire application into a single distributable executable anymore - heck even the tooling barely exists anymore. But instead of solving problems like dependency management and linking, today's engineers simply build piles of abstraction into the problem space until the thing you want to do more than anything (i.e. execute an application) becomes a single call. Of course you now need to build and maintain those abstract towers, so more jobs for everybody!

gorfian_robot 1y ago
this is what happens when hw is too cheap
- zbentley 1y ago
  You sure? Which hardware? Put another way: stuff like Electron makes a pretty good case for the "cheap hardware leads to shitty software quality/distribution mechanisms" claim. But does Docker? Containers aren't generally any more expensive in hardware other than disk-space to run than any other app. And disk space was always (at least since the advent of the discrete HDD) one of the cheapest parts of a computer to scale up.
  - wmf 1y ago
    If you go back to the Sun days, you literally could not afford enough servers to run one app per server so instead you'd hire sysadmins to figure out how to run Sendmail and Oracle and whatever on one server without conflicting. Then x86/Linux 1Us came out and people started just running one app per server ("server sprawl") which was easy because there was nothing to conflict. This later became VM sprawl and containers were an optimization on that.
    - zbentley 1y ago
      I'm not getting it, sorry. We had to have multiple apps per server before, and now we have containers which offer a convenient way to have multiple apps per server? That seems like the same thing. Could you explain more re: what you meant?

wucke13 1y ago

My take: containers forced devepopers to declare various aspects of the application in a standardized, opinioated way: - Persistant state? Must declare a volume. - IO with external services? Must declare the ports (and maybe addresses). - Configurable parameters? Must declare some env variables. - Trasitive dependecies? Must declare them, but using a mechanism of your choosing (e.g. via the package manager of your base image distro). Separation of state (as in persistency) and application (as in binaries, assets) makes updates easy. Backups also. Having most all IO visible and explicit simplifies operation and integration. And a single, (too?!?) simple config mechanism increases reusability, by enabling e.g. lightweight tailoring of generic application service containers (such as mariadb). Together this bunch of forced, yet leaky abstractions is just good enough to foster immense reuse & composability on to a plethora of applications, all while allowing to treat them almost entirely like blackboxes. IMHO that is why OCI containers became this big, compared to other virtualization and (application-) cuntainer technologies.

ranger207 1y ago
Yep, this is why. Containers are a way to package more of the OS environment than you can do otherwise
lotyrin 1y ago
Yeah. I remember when "okay, cool, how do we deploy, run, maintain the thing we (our leadership) just bought from you" was several weeks of meetings with the vendor and discussing awful "human-readable" documentation (if it existed). Those meetings still happen sometimes, but "cloud" *aaS and containers really put a dent in that sort of thing.

0xbadcafebee 1y ago

Why it happened is not nearly as important as what it unveiled: that versioned immutable systems are the most powerful system design concept in history. Most people have not yet grasped what an insanely powerful concept it is. At some point in the future, maybe 50-100 years from now, someone will look back and say "holy shit; deploying web apps are where this concept came from?" I hope in my lifetime that people get it and start applying it to other system designs/components.

hackshack 1y ago
radmind had this philosophy. https://radmind.org https://radmind.org It let you create diffs of a filesystem, and layer them with configurations, similar to containers. Useful for managing computer labs at the time.
- 0xbadcafebee 1y ago
  Well disk (or tape) images are as old as 'dd' and 'tar', that's not the revolutionary part. If the disk is writeable the state is still constantly mutating, so you're fighting a war of attrition (it's configuration management at that point, which is terrible). But a read-only disk that doesn't accrue changes, and only needs a reboot to fix, that's the revolutionary part. Anybody who ran thin-terminals can tell you how reliable and easier to manage those are than a full-blown OS. At some point in the future people are going to realize that every system should work that way.

gorfian_robot 1y ago

in some situations, the cloud is a no-go and the lead time to install new hardware is quite lengthy. in those case vm's or containers allow for rapid change at the software/OS level while the meat space still moves like a glacier. sounds minor, but it is a Big Deal for some

nkozyra 1y ago

> was always surprised someone didn't invent a tool for ftping to your container and updating the PHP. No FTP needed, you can just mount the application directory.

daxfohl 1y ago

> Application composition from open source components became the dominant way of constructing applications over the last decade. I'm just as interested in why this ^ happened. I imagine it's pretty unique to software? I don't hear of car companies publishing component designs free for competitors to use, or pharmaceuticals freely waiving the IP in their patents or processes. Certainly not as "the dominant way of" doing business. I wonder if LLM coding assistants had come about earlier, whether this would have been as prevalent. Companies might have been more inclined to create more of their own tooling from scratch since LLMs make it cheap (in theory). Individuals might have been less inclined to work on open source as hobbies because LLMs make it less personal. Companies might be less inclined to adopt open-source LLM-managed libraries because it's too chaotic.

travisjungroth 1y ago
I think open source software took off because it’s more standalone than the other things you listed and this makes the rewards much higher. If I write some code, it needs a computer and environment to run. If I’m writing for what’s popular, that’s pretty much a given. In short, for code the design is the product. If I design a pharmaceutical, someone still has to make it. Same for car parts. This effort is actually greater than the effort of design. If you include regulation, it’s way higher. So, this great feedback loop of creation-impact-collaboration never forms. The loop would be too big and involve too much other stuff. The closest thing isn’t actually manufacturing, it’s more like writing and music. People have been reusing each other’s stuff forever in those spaces.

somat 1y ago

containers happened because the original execution isolation environment(the process) was considered a lost cause, Processes shared too much with each other so additional isolation primitives had to be added, but they had to be sort of tacked on to the side because more important than security or correctness is backwards compatibility. so now containers are considered a different thing than processes when really they are process with these additional isolation primitives enabled.

ghaff 1y ago
In the early 2000s (yes, long after the original jails), containers were pitched as an alternative to VMware's VMs. They lost out for a variety of reasons--but mostly because as purely a different (and somewhat lighter-weight) encapsulation technique they weren't that interesting.
- deleted 1y ago
  [deleted]

forrestthewoods 1y ago

Containers were invented because Linux is incapable of reliably running software programs. The global pool of shared libraries is an abject failure and containers are a heavy handed workaround.

crummy 1y ago

> Docker also made Go credible as a programming language, Can someone explain why Docker was beneficial for Go?

jopsen 1y ago
Because docker was written in golang. And before docker, not many large applications were.
- freetonik 1y ago
  Ironically, also because Go is one of few popular languages for web applications that can produce a single executable binary and does not require a container to deploy with ease.
- CamouflagedKiwi 1y ago
  I think there's a pretty big citation needed on that part of the article. I'm not clear that Docker contributed to that anywhere near as much as a general increase of momentum around Go as it became better known in the industry.
nitwit005 1y ago
I've never heard of anyone basing their choice to use Go off of the fact that Docker used it. Certainly, the fact that Google was pushing Go mattered far more. I think you should see this article as someone who's a huge fan of Docker due to working there for years.

coppsilgold 1y ago

When you launch a container (either through docker you manually through namespaces) you are effectively representing yourself to the kernel as a separate thing. This allows you to construct a completely separate environment when interacting with the kernel where none of your concerns are going to leak out and nothing you don't care for is going to leak in. When people say that static executables would solve the problem they are wrong, a static executable just means that you can eschew constructing a separate file-system inside your container - and you will probably need to populate some locations anyway. Properly configured containers are actually supposed to be secure sandboxes, such that any violation is a kernel exploit. However the Linux kernel attack surface is very large so no one serious who offers multi-tenant hosting can afford to rely on containers for isolation. They have to assume that a container escape 0day can be sourced. It may be more accurate to say that a general kernel 0day can be sourced since the entire kernel surface area is open for anyone to poke. seccomp can mitigate the surface area but also narrow down the usefulness.

112233 1y ago
not .. really. Linux kernel has no concept of a container, you have to be super careful to avoid "mixing" host stuff in. I'm yet to see an case where "leaking in" would be prevented by default. Docker "leaks in" as much as you want. Containers also do not nest gracefully (due to, e.g., uids), so cannot be used as a software component. It's mostly a linux system admin thing right now.
- coppsilgold 1y ago
  The Linux kernel provides the namespaces... Docker has made some strange decisions for default behavior but if you take a more hands on approach such as with bubblewrap/bwrap nothing will leak in.
  - 112233 1y ago
    How would you do it? I'm quite interested! How can you hide container processes in host procfs using bwrap? And make sure no mounts stay mounted in the host? The most "nothing leaks in" runtime I've seen is gVisor (before going VM). Attaining that with bwrap would be nice, but I'm sceptical.

lloeki 1y ago

> There was one key innovation, which was Docker Hub Vagrant has had that for VMs long before. Dockerfiles and Docker Hub are directly inspired from Vagrantfiles and the Vagrant Box library

01HNNWZ0MV43FF 1y ago

I just wanted to have two instances of two versions of Postgres installed, and have their data directories documented in a YAML file, and know that they aren't gonna fuck up anything else on my system if someone gets RCE or something odd

ksk23 1y ago

I love boring.

w10-1 1y ago

This and the comments may miss the forest for the trees. Enterprise software vendors sold libraries and then "application servers", essentially promising infrastructure (typically tied to databases). Enterprise software developers -- Google in particular -- got tired of depending on others' licensed infrastructure. This birthed Spring and Docker, splitting the market. (Fun aside: when is a container a vm? When it runs via Apple containerization.)

cyberax 1y ago

The article is just wrong. Before Docker, there was OpenVZ and Virtuozzo. They were used to provide cheaper "dedicatd machine" hosting back around 2005. Then the technology from OpenVZ slowly made its way into the mainline Linux, in the form of cgroups and namespaces. LWN called it a "container puzzle", with tens of moving pieces. And it was largely finished by early 2010-s. I built my own container system in 2012 that used cgroups to oversubscribe the RAM, with simple chroot-based file namespaces for isolation. We even used XFS projects (raise your hand if you know what this is!) for the disk quota management. I remember that I had to use systemtap to patch the kernel to be able to find out which process died as a result of the OOM killer, there were no standard ways to do that. We sold it as a part of our biotech startup to Illumina. Then we sold it again to Amazon as a part of another startup :) The genius of Docker was the layered overlayfs-based image building. This one simple innovation made it possible to build images in a constructive way, without having to waste half an hour for each minor change. I was floored with its simplicity and power when I first saw it.

man8alexd 1y ago
OpenVZ didn't use cgroups but its own resource limits, which sucked.
- cyberax 1y ago
  Well, yes. "Beancounters" and it also was not nestable. That's why it took quite a bit of time to get it all upstreamed.
  - man8alexd 1y ago
    Unless my memory fails me, OpenVZ was never upstreamed.

Joker_vD 1y ago

> What people do with Docker is spin up a database or another service to develop or test against. Yep. Being able to run docker run --rm --publish=127.0.0.1:27017:27017 'mongo:3.6.8' or docker run --rm --publish=127.0.0.1:27017:27017 'mongo:5.0' and then get rid of it with simple Ctrl-C is a godsend.

DanielHB 1y ago
I remember installing Oracle in my desktop computer to be able to do my programming classes... If the installation crashed (which was quite common, happened to me once) it was easier to just format the computer completely and start again. Effed up the database? Probably easier to format everything and install from scratch

dilyevsky 1y ago

Linux CGroups specifically were started at Google because their cluster management system Borg (or maybe it was still Babysitter at the time) needed a way to do resource tracking and admission control. Here's a comment by one of original devs: https://news.ycombinator.com/item?id=25017753 https://news.ycombinator.com/item?id=25017753

deviation 1y ago
This is also somewhat highlighted in Google's paper "Borg, Omega, and Kubernetes" which they published in 2016. https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/44843.pdf https://static.googleusercontent.com/media/research.google.c...

boomskats 1y ago

Fwiw the actual video that he links to is well worth a watch. Featuring one of the most Justin intros ever.

alexey-salmin 1y ago

The article seems to assume that containers appeared to solve the software distribution problem and then somehow got repurposed into virtualization, isolation and management of production services. I think this view is very far from truth. The virtualization/isolation aspect came first, the SWSoft Virtuozzo was doing that quite well in early 2000s. They even had [some] IO isolation which I think took around a decade to support elsewhere. Then gradually pieces of Virtuozzo/OpenVZ reached the mainline in a form of cgroups/LXC and the whole thing slowly brewed for a while until the Docker added the two missing pieces: the fast image rebuilds and the out-of-the-box user experience. Docker of course was the revolution, but by then sufficiently advanced companies have been already using containers for isolation for a full decade.

bayindirh 1y ago
I believe BSD Jails created a "Why don't we have this under Linux" feeling as well.
rmoriz 1y ago
To my knowledge the very first appearance was the "ctx" kernel patches project by linux-vserver.org by Herbert Pötzl et al. https://web.archive.org/web/20250307173707/http://linux-vserver.org/Welcome_to_Linux-VServer.org https://web.archive.org/web/20250307173707/http://linux-vser... (sadly, the domain seems to be expired just recently) I used it heavily back in the days, long before kernel namespaces went upstream and docker became a thing.
DanielHB 1y ago
I remember hearing about all the support from Intel for "0-cost" virtualization at the hardware level way before I heard about containers. From what I remember it was mostly to speed up virtual machines (VMWare stuff). It was a massive market differentiator for Intel in the server space. I vaguely remember having to turn on some features in VirtualBox at the time to speed up my VMs, it was a massive uplift in performance if you had a CPU that supported it.

kotaKat 1y ago

Containers happened because nobody knew what the hell they were doing and still have no clue what the hell they are doing. Software by the deranged for the deranged.

goodpoint 1y ago
...and they don't care about security.

0dayz 1y ago

What blew my mind and convinced me to only use immutable distros is the immutability of it. For instance I could create my own login screen for an web service without having to worry about the package manager overriding my code, because I inject it into the container, which is already updated. I can also forcefully reroute much easier ports or network connections the way I want it.

lazylizard 1y ago

how did solaris zones get left out of the story?

lproven 1y ago
George Santayana and Henry Spencer can tell you. Santayana in 1905: "Those who cannot remember the past are condemned to repeat it." Spencer in 1987: "Those who do not understand Unix are condemned to reinvent it, poorly."

bboreham 1y ago

The tool that connects into your container and updates the files was called "Tilt".

nahuel0x 1y ago

Because containers were lighter than VM's, made reproducible and modularizable how to build the VM contents, and "solved" Unix/POSIX process isolation / library management design flaws.

yohbho 1y ago

> Docker was easy to adopt as it did not change very much about how you used software. What?! Docker images should seperate the program from the data. Something casual image maintainers fail to adhere to. I find docker extreme hard to use for persistent state that is not a remote DB. It is even less secure than VMs (which seem to be like swiss chess if you try to break out). (Security-wise: rkt failed five years ago, BSD jails could do it better?) Also, Docker was supposed to solve performance issues of VMs, debloating them. Instead, we got hypervisors or full VMs as hosts for Docker images, just to get the image as a de facto meta package manager to ship software. So for me, it can be convenient, but I have no idea what I am doing when adding make file like statements to a yml to run full containers just to run e.g. a web server with some dependecies. But the workflow for that was novell to me, who previously fought with ruby and python module installations on host system.

lproven 1y ago

Money. Containers were around for a decade or more on FreeBSD and Solaris. They let you divvy up expensive big Unix iron. Same as VMs were around on mainframes from the late 1960s and expensive Unix RISC servers from the late 1980s. Linux didn't need it because it was cheap. So Linux replaced that older more expensive stuff, on cheap COTS hardware: x86. Once everything was commoditised and cost-cut, suddenly, efficiency started to matter, so VMware thrived and was copied and VMs were everywhere. Then the low usage and inefficiency of resource sharing of VMs made them look expensive, so they started to get displaced by the cheaper easier tech of containers, making "it works on my machine, so let's ship my machine" scale to production.

pjmlp 1y ago
Even earlier, I was introduced to the concept of containers in HP-UX Vault back in 1999, before FreeBSD and Solaris got them. Unfortunely HPe has removed most of HP-UX documentation out of Internet so it is hard to point it out. However there are still some digital traces, HP-UX 10.24 release, > This is a Virtual Vault release of HP-UX, providing enhanced security features. Virtual Vault is a compartmentalised operating system in which each file is assigned a compartment and processes only have access to files in the appropriate compartment and unlike most other UNIX systems the superuser (or root) does not have complete access to the system without following correct procedures. https://en.wikipedia.org/wiki/HP-UX https://en.wikipedia.org/wiki/HP-UX A forum discussion, https://community.hpe.com/t5/operating-system-hp-ux/hp-virtual-vault/td-p/3865588 https://community.hpe.com/t5/operating-system-hp-ux/hp-virtu... Now it would be great to get back those HP-UX Vault PDFs.
- lproven 1y ago
  Fair enough. I never worked on HP/UX. AIX, Solaris, Xenix, and VMS, OS/400, and RSTS, but never HP. I wrote about containers being the next big thing in 2011: https://www.theregister.com/Print/2011/07/18/brief_history_of_virtualisation_part_3/ https://www.theregister.com/Print/2011/07/18/brief_history_o... I credit FreeBSD and Solaris and AIX, but I think I didn't know about HP/UX.
  - 1y ago

rdsubhas 1y ago

I used cgroups, lxc, chroots, self-extracting executables. I built rugged, portable applications for UNICEF laptops and camps before docker was a thing. And I think this whole point about "virtualization", "security", making the most use of hardware, reducing costs, and so on, while true, it's an "Enterprise pitch" targeted at heads of tech and security. Nice side effects, but I couldn't care less. There are real, fundamental benefits to containers for a solo developer running a solo app on a solo server. Why? My application needs 2 or 3 other folders to write or read files into, maybe 2 or 3 other runtime executables (jvm, node, convert, think of the dozens of OSS CLI tools, not compile-time libraries), maybe apt-get install or download a few other dependencies. Now I, as a indie developer, can "mkdir" a few files from a shell script. But that "mkdir" will work the first time. It will fail the second time saying "directory already exists". I can "apt-get install" a few things, but upgrading and versioning is a different story altogether. It's a matter of time before I realize I need atleast some barebones ansible or state management. I can tell you many times how I've reinvented "smallish" ansible in shell scripts before docker. Now if I'm in an enterprise, I need to communicate this entire State of my app – to the sysadmin teams. Forget security and virtualization and all that. I need to explain every single part of the state, versions of java and tomcat, the directories, and all those are moving targets. Containers reduce state management. A LOT. I can always "mkdir". I can always "apt-get install". It's an ephemeral image. I don't need to write half-broken shell scripts or use ansible or create mini-shell-ansible. If you use a Dockerfile with docker-compose, you've solved 95% of state management. The only 5% left is to docker-compose the right source. Skip the enterprisey parts. A normal field engineer or solo developer, like me, who's deploying a service on the field, even on my raspberry pi, would still use containers. It boils down to one word: "State management" which most people completely underestimate as "scripting". Containers grant a large control on state management to me, the developer, and simplify it by making it ephemeral. That's a big thing.

jiggunjer 1y ago
That's two words. How about "deterministic".
- abound 1y ago

superkuh 1y ago

Pretty simple. Modern development cycles for the libs that software depends on now happen significantly faster that OS/Distros update. This is future shock. The symptoms of this disease express as containers. Do your part. Don't code on the bleeding edge. Keep things stable for users.

marsven_422 1y ago

[dead]

stego-tech 1y ago

A really good post that captures the IT perspective of infrastructure change and modernization over my career. They also hit the nail on the head that one of today's problems is itself automation, where we've built tools that let us create sprawling estates with no accountability for cost other than an opaque monthly bill, though I'd also chip in that said automation is often restricted to whoever your chief cloud provider is (AWS, Azure, etc). Where I find myself advocating today is very much a "rational check" on infrastructure, and curtailing accordingly. We have the tooling to ensure high availability, but does everything need to be HA? Do our SLAs for enterprise tooling really need five-nines of availability, or can we knock some applications down to a limited schedule? Does dev/test need to be live 24/7, or can we power it off when not in use? Why are we only focusing on availability and not scalability? The list goes on, but they're also not popular in enterprises with entrenched politics, which admittedly is where I find myself struggling against the current. If my social chops were better, I suspect I'd thrive in consultancy doing just that. All that being said, I do like containers that are done right (properly documented, secure-by-default, ready for scaling), and I continue driving more applications towards containerization in the enterprise where possible. They're the right solution for ~60-80% of enterprise use cases, with the difficulty being getting vendors on board with the idea that their software won't have a dedicated VM or hardware anymore (which everyone fiercely resists, because container-based licensing can be a PITA to them). For the rest, VMs are more than fine, and we have a growing number of ways for both to exist peacefully in the same environment. As this area of technology matures (along with "backporting" from hyperscalers to private cloud again) further, I'm really looking forward to managing global estates in smaller teams for bigger firms - things that VMs, Containers, and Infrastructure-as-Code allow.

sharts 1y ago

Why does every discussion about containers always end up revolving around linux and cgroups? Didn’t we have Zones and even Jails way back?

majke 1y ago

Most of the responses here are knee-jerk. I liked the article. It's close to my adverntures with containers. I think the invention of docker is indeed mostly packaging: Dockerfile (functional) is pretty neat, docker hub (addressing a container) is awesome, and the ENTRYPOINT in Dockerfile is great, it distinguishes Docker from .deb. But indeed, beyond Dockerfile things are bleak. Docker compose never rose to my expectations. To get serious things you need load blancer, storage, addressing, and these are beyond traditional containers scope.

Saris 1y ago

I have no idea how close I am, but Linux seems to have no way of having multiple versions of a dependency at once like Windows does. So I suspect thats the main reason.