6 ms·

My headcanon is that Docker exists because Python packaging and dependency management was so bad that dotCloud had no choice but to invent some porcelain on top

My headcanon is that Docker exists because Python packaging and dependency management was so bad that dotCloud had no choice but to invent some porcelain on top of Linux containers, just to provide a pleasant experience for deploying Python apps.

frumplestlatz 1y ago
Pretty much this; systems with coherent isolated dependency management, like Java, never required OS-level container solutions. They did have what you could call userspace container management via application servers, though.
- drowsspa 1y ago
  NodeJS, Ruby, etc also have this problem, as does Go with CGO. So the problem is the binary dependencies with C/C++ code and make, configure, autotools, etc... The whole C/C++ compilation story is such a mess that almost 5 decades ago inventing containers was pretty much the only sane way of tackling it. Java at least uses binary dependencies very rarely, and they usually have the decency of bundling the compiled dependencies... But it seems Java and Go just saw the writing on the wall and mostly just reimplement everything. I did have problems with the Snappy compression in the Kafka libraries, though, for instance .
  - skydhash 1y ago
    The issue is with cross platform package management without proper hooks for the platform themselves. That may be ok if the library is pure, but as soon as you have bindings to another ecosystem (C/C++ in most cases), then it should be user/configurable instead of the provider doing the configuration with post installs scripts and other hacky stuff. If you look at most projects in the C world, they only provide the list of dependencies and some build config Makefile/Meson/Cmake/... But the latter is more of a sample and if your platform is not common or differs from the developer, you have the option to modify it (which is what most distros and port systems do). But good luck doing that with the sprawling tree of modern packages managers. Where there's multiple copies of the same libraries inside the same project just because.
- antonvs 1y ago
  I don't agree with this. Java systems were one of the earliest beneficiaries of container-based systems, which essentially obsoleted those ridiculously over-complicated, and language-specific, application servers that you mentioned.
  - lmm 1y ago
    Java users largely didn't bother with containers IME, largely for the same reasons that most Java users didn't bother with application servers. Those who did want that functionality already had it available, making the move from an existing Java application server to Docker-style containers a minor upgrade at best.
    - antonvs 1y ago
      This is just a testament to how widely Java is used, and in how many different ways. Sounds like you're more focused on "Core Java" applications, or something like that. Every company I've been with since the late 90s was using application servers of some kind until Docker came along. And all of the more recent ones switched to containers and ditched the application servers. The switch was often much more than a minor upgrade, because it often made splitting up monoliths possible in ways that the Java ecosystem itself didn't have good support for.
  - ruszki 1y ago
    Tomcat and Jetty are application servers which are in almost every Spring application. There are such application servers which you mentioned, like Wildfly, but they are not obsolete as a whole.
    - antonvs 1y ago
      Tomcat and Jetty are not application servers according to the Jakarta EE definition. They're just servlet containers. The reason Spring includes those libraries is partly historical - Spring is old, and dates from the applications server days. Newer frameworks like Micronaut and Quarkus use more focused and performant libraries like Netty, Vert.x, and Undertow instead.
- theamk 1y ago
  Funnily enough, one of the first containers I did on my current job was to package a legacy Java app. It was pretty old, and required a very specific version of java, not available on modern systems. Plus some config files in global locations. Packaging it in the docker container made it so much easier to use.
  - p2detar 1y ago
    That makes sense, but still - there is nothing OS-specific here, like system lib or even a database, it's just the JRE version that you needed to package into the container, or am I missing something?
IshKebab 1y ago
Exactly this, but not just Python. The traditional way most Linux apps work is that they are splayed over your filesystem with hard coded references to absolute paths and they expect you to provide all of their dependencies for them. Basically the Linux world was actively designed to apps difficult to distribute.
- dabockster 1y ago
  > Basically the Linux world was actively designed to apps difficult to distribute. It has "too many experts", meaning that everyone has too much decision making power to force their own tiny variations into existing tools. So you end up needing 5+ different Python versions spread all over the file system just to run basic programs.
- antod 1y ago
  It wasn't about making apps difficult to distribute at all, that's a later side effect. Originally distros were built around making a coherent unified system of package management that made it easier to manage a system due to everything being built on the same base. Back then Linux users were sysadmins and/or C programmers managing (very few) code dependencies via tarballs. With some CPAN around too. For a sysadmin, distros like Debian were an innovative godsend for installing and patching stuff. Especially compared to the hell that was Windows server sysadmin back in the 90s. The developer oriented language ecosystem dependency explosion was a more recent thing. When the core distros started, apps were distributed as tarballs of source code. The distros were the next step in distribution - hence the name.
  - IshKebab 1y ago
    Right but those things are not unrelated. Back in the day if you suggested to the average FOSS developer that maybe it should just be possible to download a zip of binaries, unzip it anywhere and run it with no extra effort (like on Windows), they would say that that is actively bad. You should be installing it from a distro package!! What about security updates of dependencies?? And so on. Docker basically overrules these impractical ideas.
ecnahc515 1y ago
Sure they definitely were using Docker for their own applications, but also dotCloud was itself a PaaS, so they were trying to compete with Heroku and similar offerings, which had buildpacks. The problem is/was that buildpacks aren't as flexible and only work if the buildpack exists for your language/runtime/stack.
bane 1y ago
That's basically correct. But the more general problem is that engineers simply lost the ability to succinctly package applications and their dependencies into simple to distribute and run packages. Somehow around the same time Java made .jar files mainstream (just zip all the crap with a manifest), the rest of the world completely forgot how to do the equivalent of statically linking in libraries and that we're all running highly scheduled multithreaded operating systems now. The "solution" for a long time was to spin up single application Virtual Machines, which was a heavy way to solve it and reduced the overall system resources available to the application making them stupidly inefficient solutions. The modern cloud was invented during this phase, which is why one of the base primitives of all current cloud systems is the VM. Containers both "solved" the dependency distribution problem as well as the resource allocation problem sort of at once.
- procaryote 1y ago
  > engineers simply lost the ability to succinctly package applications and their dependencies into simple to distribute and run packages. but this is what docker is If anything, java kinda showed it doesn't have to suck, but as not all things are java, you need something more general
  - BSVogler 1y ago
    With the difference that with docker you are shipping the runtime to your source code as well.
    - arccy 1y ago
      which is great when you realize that not all software is updated at the same time. how managing multiple java runtime versions is supposed to work is still beyond me... it's a different tool at every company, and the instructions never seem to work
      - qcnguy 1y ago
        Java runtimes is just: export JAVA_HOME=/path; ./app.sh
globular-toast 1y ago
Pyinstaller predates Docker. It's not about any individual language not being able to do packaging, it's about having a uniform interface for running applications in any language/architecture. That's why platforms like K8s don't have to know a thing about Python or anything else and they automatically support any future languages too.

frumplestlatz 1y ago

Pretty much this; systems with coherent isolated dependency management, like Java, never required OS-level container solutions. They did have what you could call userspace container management via application servers, though.

drowsspa 1y ago
NodeJS, Ruby, etc also have this problem, as does Go with CGO. So the problem is the binary dependencies with C/C++ code and make, configure, autotools, etc... The whole C/C++ compilation story is such a mess that almost 5 decades ago inventing containers was pretty much the only sane way of tackling it. Java at least uses binary dependencies very rarely, and they usually have the decency of bundling the compiled dependencies... But it seems Java and Go just saw the writing on the wall and mostly just reimplement everything. I did have problems with the Snappy compression in the Kafka libraries, though, for instance .
- skydhash 1y ago
  The issue is with cross platform package management without proper hooks for the platform themselves. That may be ok if the library is pure, but as soon as you have bindings to another ecosystem (C/C++ in most cases), then it should be user/configurable instead of the provider doing the configuration with post installs scripts and other hacky stuff. If you look at most projects in the C world, they only provide the list of dependencies and some build config Makefile/Meson/Cmake/... But the latter is more of a sample and if your platform is not common or differs from the developer, you have the option to modify it (which is what most distros and port systems do). But good luck doing that with the sprawling tree of modern packages managers. Where there's multiple copies of the same libraries inside the same project just because.
antonvs 1y ago
I don't agree with this. Java systems were one of the earliest beneficiaries of container-based systems, which essentially obsoleted those ridiculously over-complicated, and language-specific, application servers that you mentioned.
- lmm 1y ago
  Java users largely didn't bother with containers IME, largely for the same reasons that most Java users didn't bother with application servers. Those who did want that functionality already had it available, making the move from an existing Java application server to Docker-style containers a minor upgrade at best.
  - antonvs 1y ago
    This is just a testament to how widely Java is used, and in how many different ways. Sounds like you're more focused on "Core Java" applications, or something like that. Every company I've been with since the late 90s was using application servers of some kind until Docker came along. And all of the more recent ones switched to containers and ditched the application servers. The switch was often much more than a minor upgrade, because it often made splitting up monoliths possible in ways that the Java ecosystem itself didn't have good support for.
- ruszki 1y ago
  Tomcat and Jetty are application servers which are in almost every Spring application. There are such application servers which you mentioned, like Wildfly, but they are not obsolete as a whole.
  - antonvs 1y ago
    Tomcat and Jetty are not application servers according to the Jakarta EE definition. They're just servlet containers. The reason Spring includes those libraries is partly historical - Spring is old, and dates from the applications server days. Newer frameworks like Micronaut and Quarkus use more focused and performant libraries like Netty, Vert.x, and Undertow instead.
theamk 1y ago
Funnily enough, one of the first containers I did on my current job was to package a legacy Java app. It was pretty old, and required a very specific version of java, not available on modern systems. Plus some config files in global locations. Packaging it in the docker container made it so much easier to use.
- p2detar 1y ago
  That makes sense, but still - there is nothing OS-specific here, like system lib or even a database, it's just the JRE version that you needed to package into the container, or am I missing something?

IshKebab 1y ago

Exactly this, but not just Python. The traditional way most Linux apps work is that they are splayed over your filesystem with hard coded references to absolute paths and they expect you to provide all of their dependencies for them. Basically the Linux world was actively designed to apps difficult to distribute.

dabockster 1y ago
> Basically the Linux world was actively designed to apps difficult to distribute. It has "too many experts", meaning that everyone has too much decision making power to force their own tiny variations into existing tools. So you end up needing 5+ different Python versions spread all over the file system just to run basic programs.
antod 1y ago
It wasn't about making apps difficult to distribute at all, that's a later side effect. Originally distros were built around making a coherent unified system of package management that made it easier to manage a system due to everything being built on the same base. Back then Linux users were sysadmins and/or C programmers managing (very few) code dependencies via tarballs. With some CPAN around too. For a sysadmin, distros like Debian were an innovative godsend for installing and patching stuff. Especially compared to the hell that was Windows server sysadmin back in the 90s. The developer oriented language ecosystem dependency explosion was a more recent thing. When the core distros started, apps were distributed as tarballs of source code. The distros were the next step in distribution - hence the name.
- IshKebab 1y ago
  Right but those things are not unrelated. Back in the day if you suggested to the average FOSS developer that maybe it should just be possible to download a zip of binaries, unzip it anywhere and run it with no extra effort (like on Windows), they would say that that is actively bad. You should be installing it from a distro package!! What about security updates of dependencies?? And so on. Docker basically overrules these impractical ideas.

ecnahc515 1y ago

Sure they definitely were using Docker for their own applications, but also dotCloud was itself a PaaS, so they were trying to compete with Heroku and similar offerings, which had buildpacks. The problem is/was that buildpacks aren't as flexible and only work if the buildpack exists for your language/runtime/stack.

bane 1y ago

That's basically correct. But the more general problem is that engineers simply lost the ability to succinctly package applications and their dependencies into simple to distribute and run packages. Somehow around the same time Java made .jar files mainstream (just zip all the crap with a manifest), the rest of the world completely forgot how to do the equivalent of statically linking in libraries and that we're all running highly scheduled multithreaded operating systems now. The "solution" for a long time was to spin up single application Virtual Machines, which was a heavy way to solve it and reduced the overall system resources available to the application making them stupidly inefficient solutions. The modern cloud was invented during this phase, which is why one of the base primitives of all current cloud systems is the VM. Containers both "solved" the dependency distribution problem as well as the resource allocation problem sort of at once.

procaryote 1y ago
> engineers simply lost the ability to succinctly package applications and their dependencies into simple to distribute and run packages. but this is what docker is If anything, java kinda showed it doesn't have to suck, but as not all things are java, you need something more general
- BSVogler 1y ago
  With the difference that with docker you are shipping the runtime to your source code as well.
  - arccy 1y ago
    which is great when you realize that not all software is updated at the same time. how managing multiple java runtime versions is supposed to work is still beyond me... it's a different tool at every company, and the instructions never seem to work
    - qcnguy 1y ago
      Java runtimes is just: export JAVA_HOME=/path; ./app.sh

globular-toast 1y ago

Pyinstaller predates Docker. It's not about any individual language not being able to do packaging, it's about having a uniform interface for running applications in any language/architecture. That's why platforms like K8s don't have to know a thing about Python or anything else and they automatically support any future languages too.