7 ms·

If you use chroot to run something, it's interesting how the dynamic libs you need to get in place grows until you are mirroring a whole linux in a subtree. It

If you use chroot to run something, it's interesting how the dynamic libs you need to get in place grows until you are mirroring a whole linux in a subtree. It gives you a sense for how you end up with containers. One thing that is wild to me is how nix solves this problem, of things needing to be linked together. It doesn't solve it with containers, but by rewriting the location of the links in the executable to be in the nix store. You can run LDD and see it in action. To me, all that points at containers being in some way a solution to Dynamic linking. And maybe an over the top solution. Should we be doing more static linking? Not even depending on libc? What are the challenges with that?

tutfbhuf 3y ago
One issue with static linking is that your dependencies will likely have critical CVEs over time. If you keep all your libraries separate on the filesystem, you can just do a "apt update; apt upgrade", and you will have all the latest patches. This will patch security issues in e.g. libssl or libc for all your applications that are dynamically linked against this shared libraries, which can be quite a few. In static binaries, the version of the libraries is not obvious from the outside. If you have, for example, 100 fully static binaries, these can come in 100 different major/minor/patch level versions of their dependencies. You now have to patch each binary separately by upgrading and recompilation 100 times to patch all your static binaries, that requires much more time and energy.
- adamgordonbell 3y ago
  That all makes sense. But when those 100 binaries end up as 100 OCI images, and then to patch them you need to update those 100 OCI images to have the new version, it does seem like we've gone in a circle a bit. I mean, there are some advantages, if they all share the same base layer, maybe they share those libs at least on disk via a shared layer. But practically, though you are maybe not back where you started, you are at a place that seems to share some similarities.
  - nurple 3y ago
    This^ This is one of the biggest issues with containers IMO. This and the layering system, which I think is poorly designed both to configure and to actually do the tasks it's meant to(build and delivery caching). The solutions to this problem in the space have basically been to provide scanners to crack open the containers and detect things with known vulnerabilities. But I have not seen (m)any solutions around these scanners to facilitate the lifecycle of landing fixes. Even if you provided a tree of a-proved base containers for each deployment lang in your org, you can't just update the base and deploy the world, there's not even tooling to automate working over the "FROM hierarchy" of images where you could detect which need to use new bases. Because of the difficulty in managing large container hierarchies, in some orgs the later drives a common methodology of making image tags mutable, i.e. `ruby:myorg-v2`, which makes the FROM more like a dynamic link reference that gets updated automatically on the next build. I view this workaround as a regression brought on by the _still_ incredibly poor and complex SDLC tooling around managing images.
    - rdtsc 3y ago
      It's like for a few years the whole computing world forgot about security updates and went on a container bender, when woke up one morning realizing what they've done, and then started adding various clunky solutions on top: mutable tags, various tooling to take apart container images and inspect the junk in there, notify and scan for issues etc.
      - pjmlp 3y ago
        You mean like proposing JSON as XML replacement, rebuilding from scratch all the XML tooling including validation, and when almost done with it, replacing JSON with YAML?
        rdtsc 3y ago
        Exactly! YAML apparently is now terrible as well, but not sure what's next in the pipeline, toml perhaps? I think it would be funny if we reverted back to s-expressions and started the cycle all over.
nurple 3y ago
Containers are a solution to dependency management for sure. In the world of the FHS, the dynamic linker is meant to solve a number of problems, like space saving and security updates, at the OS level by discovering dynamic deps installed in special library paths. One thing I've never liked about FHS is how everything is organized by kind. The interesting thing about how Nix approaches the problem is to replace the concept of FHS almost entirely (only a couple binaries are linked to /) by hijacking PATH, and the linker configs like you mentioned. The biggest difference being that the whole version-pinned dep tree is encoded in a nix package(and in the linker config of the binaries it produces) rather than just the package itself. At some level you could say there is no "dynamic" runtime linking in nix, i.e the linker uses partially specified deps in a discovery phase, all of the link bindings happen at build time. The FHS did attempt to solve the issue of multi-version dependencies with an interesting name and symlink setup, but they are usually still bound by fairly loose version constraints (like major version). Containers are a lot more like nix in this way, where deps are "resolved" at build time by the distro's package manager by virtue of controlling the process' filesystem. This is one major issue with the reproducibility of container builds, the distro package managers are not deterministic, you could run a build back-to-back and get different deps depending on your timing(yes, even between test and build CI steps).
jayd16 3y ago
Containers are powerful because they solve many computing issues, one of which being able to act as a (lower case c) static container for dynamically linked apps as well as cross-language, multi-executable meta-apps. Containers also provide many forms of isolation (network, file system etc.), they provide a modern versioning and distribution scheme, composibility (use another container as a base image). All of these things can, and perhaps should be, done at a the language level as well but containers also work across languages, across linking paradigms, and with existing binaries.
nonameiguess 3y ago
I haven't seen any other response mention it yet, but containers are also heavily used for web-exposed services in part because of address space and port contention. Network namespaces allow you to graft an overlay network onto your physical network in a relatively simple and easy way (not that it's actually easy, but networking never is). Otherwise, sure, nix can rewrite the RPATH in your ELF file to make it pull dynamic libs from the nix store, but what does it do when two processes both want to listen on ports 80 and 443? Possibly, if the Internet ever actually goes pure IPv6, one LAN will have enough addresses to assign one to each process instead of each host. There are, of course, other ways to handle it. People used vhosts predominantly defined in a dedicated web server that was really only a reverse proxy, but now you need nix and nginx. Then you discover you also want resource isolation. Is there a userspace alternative to cGroups? I don't see how there could even in principle be an alternative to PID/UID namespaces and UID/GID submapping. Some things have to happen in the kernel and that means containers of some sort. It doesn't have to be the exact OCI standard that eventually grew out of Docker and eventually Kubernetes, but some kind of container.
- adamgordonbell 3y ago
  I think what you are saying is true, and my knowledge of networking is pretty slim. But, to play along with my static linking thought-exercise: if you take a process and put it in a network namespace then is it a container? I wouldn't say it is. Container runtimes might have a nice interface for namespacing, but namespacing something doesn't make it a container. I guess my thought experiment is if things are statically linked binaries and you had a way to run them with the control group and namespace settings you wanted, would the packaging aspect of containers add anything? The elites don't want you to know it, but namespaces are just there for the taking. You can grab as many as you want. You can set the memory limit on any process with cgroups, no docker desktop required. :) Anyways, just a thought experiment about how the industry sometimes seems to be going in a circle, in the fashion of the lady who swallowed a fly.
pjmlp 3y ago
That wasn't the deal in HP-UX vaults, Solaris Zones, System 360/MVS virtualization,... The way they are used on GNU/Linux is indeed a solution to GNU/Linux's software distribution issues on a highly fragmented landscape.
bayindirh 3y ago
Disclaimer: I'm not a strong containerization proponent. The good part of containers is you isolating the thing you're running. I'm very against resource waste, but if I can spend 90MB on a container image instead of installing a complete software stack to run a task which is executed weekly and runs for 10 minutes, I'd prefer that. Plus, I can create a virtual network and storage stack around the container(s) if I need to. Case in point: I use imap-backup to backup my e-mail accounts, but it's a ruby application and I need to install the whole stack of things, plus the gems. Instead I containerize it, and keep my system clean. Nix is something different and doesn't solve "many foreign processes not seeing each-other on the same OS" problem. Heck, even Docker doesn't solve all problems, so we have "user level containers" which do not require root access and designed to be run in multitenant systems.
- marcosdumay 3y ago
  Linux doesn't solve that problem at all. Qubes tries to do that with a hack, that is probably sufficient, but quite complex. I'd say that we don't have good solutions to that problem. But nobody even tried our best candidates, so I'm not completely sure. (For a while, it looked like Android would finally try some. But then Google turned it into a user-hostile anti-privacy OS.)
- adamgordonbell 3y ago
  > many foreign processes not seeing each-other on the same OS For sure, I was thinking of the packaging nature of containers, not the 'security' nature of containers. The pivotroot part. Though I guess being able to have namespaces does make packaging clearer in certain cases. For a horrible analogy: With actual shipping containers, we don't have each shipping container be a stripped down model of a ship, so that the things in it aren't confused.
  - samatman 3y ago
    Agreed, it isn't great. Shipping containers are more like tarballs: a bunch of stuff (files) bundled into one "stuff" (file) so they're easy to ship around and pieces don't get lost in the process. I don't think there is a good physical-world analogy for what containers are doing, though. Maybe takeout including utensils and napkins to "simulate a kitchen" in case you eat it on the go, so you don't have to rely on the "system" forks and napkins? Still kinda rubbish honestly.
Already__Taken 3y ago
dynamic linking to me seems like solving a problem that filesystems should. deduplicating data.
- ahepp 3y ago
  Deduplication, in the sense of physical space reduction, seems like the least strong argument one could make for dynamic linking these days. The strongest argument I can think of, is enabling system managers (distro maintainers, even end users) to update dependencies. This might be to apply a security patch, enable some kind of tracing for profiling an application, and so on.
- pjmlp 3y ago
  Plugins have nothing to do with filesystems.
- watermelon0 3y ago
  In practice, dynamic linking might not be achiving that much in the sense of deduplication: https://drewdevault.com/dynlib.html https://drewdevault.com/dynlib.html
- bayindirh 3y ago
  As a bonus, shared libraries are deduplicated in RAM, too. Plus you don't need to recompile the universe because of a patch.
- brabel 3y ago
  That's an interesting way to look at it. If all dynamic libs lived on a read-only location, then the file system could actually only store the libs in one place and the other "copies" would be just symlinks to that... and when the OS loaded such lib, it would automatically know that despite being in different locations, the libs were the same (they're all symlinks to the same place). Is this something that has been attempted before?
  - nurple 3y ago
    This is exactly how nix works, except instead of symlinks it actually modifies the binaries at buld-time to point its dynamic library paths to absolute paths in the store which includes a source-derived hash in place of a version.
silverquiet 3y ago
Funny - it looks like you’re being downvoted for asking what I think is a very natural question. It’s one I’ve asked before; have we just created a more elaborate statically-linked executable via containerization? In the end, Docker/OCI seems like the universal Linux package manager. I’m sure I don’t have the full picture since I’m far more ops than dev though.
- bayindirh 3y ago
  I don't think so. Instead we created an artifact which can live inside an OS, but cannot see and touch to the rest of the OS. CGroups is a deceptively powerful mechanism. You can isolate a process resource wise (X cores, Y amount of memory, Z amount of swap), network wise (a different virtual network adapter with its own IP, bandwidth limits, etc.) and FS wise (running in its own filesystem with devices it can see). It can keep your system tidy by encapsulating elaborate stacks which makes system management painful, allows deterministic operation and image generation if you tag everything with version. Downside is you can do bad things with it like terminating HTTPS with a gateway container and talk HTTP among your backend instead of configuring tools, or writing shoddy software, and getting away with it because it works, or gets automatically restarted when it crashes 6 times a day. I don't run every service as a container, because some services suffocate when they are in a container, but for short running things which needs system-wide changes to function, or test-driving small services before fully committing, it's a good tool to have. However, it's abused with no end, and their popular use leaves a bad taste in your mouth.
  - adamgordonbell 3y ago
    You can put any given process is a cgroup. It doesn't have to be a container. If you have a statically linked executable, setup the cgroup for it as you will. No containers needed. You can namespace is as well. No `FROM X` `RUN Y` dockerfile stuff needed.
    - bayindirh 3y ago
      > You can put any given process is a cgroup. It doesn't have to be a container. Yes, I run many programs inside cgroups, but not in containers. > If you have a statically linked executable, setup the cgroup for it as you will. No containers needed. You can namespace is as well. Yes. > No `FROM X` `RUN Y` dockerfile stuff needed. Yes. dockerfile only sets up the chroot in an overlayfs and fires up the "container" using mechanisms present in the kernel already. As I said on another comment, quoting myself: > Docker just made the interface more practical, and built the ecosystem around it.

tutfbhuf 3y ago

One issue with static linking is that your dependencies will likely have critical CVEs over time. If you keep all your libraries separate on the filesystem, you can just do a "apt update; apt upgrade", and you will have all the latest patches. This will patch security issues in e.g. libssl or libc for all your applications that are dynamically linked against this shared libraries, which can be quite a few. In static binaries, the version of the libraries is not obvious from the outside. If you have, for example, 100 fully static binaries, these can come in 100 different major/minor/patch level versions of their dependencies. You now have to patch each binary separately by upgrading and recompilation 100 times to patch all your static binaries, that requires much more time and energy.

adamgordonbell 3y ago
That all makes sense. But when those 100 binaries end up as 100 OCI images, and then to patch them you need to update those 100 OCI images to have the new version, it does seem like we've gone in a circle a bit. I mean, there are some advantages, if they all share the same base layer, maybe they share those libs at least on disk via a shared layer. But practically, though you are maybe not back where you started, you are at a place that seems to share some similarities.
- nurple 3y ago
  This^ This is one of the biggest issues with containers IMO. This and the layering system, which I think is poorly designed both to configure and to actually do the tasks it's meant to(build and delivery caching). The solutions to this problem in the space have basically been to provide scanners to crack open the containers and detect things with known vulnerabilities. But I have not seen (m)any solutions around these scanners to facilitate the lifecycle of landing fixes. Even if you provided a tree of a-proved base containers for each deployment lang in your org, you can't just update the base and deploy the world, there's not even tooling to automate working over the "FROM hierarchy" of images where you could detect which need to use new bases. Because of the difficulty in managing large container hierarchies, in some orgs the later drives a common methodology of making image tags mutable, i.e. `ruby:myorg-v2`, which makes the FROM more like a dynamic link reference that gets updated automatically on the next build. I view this workaround as a regression brought on by the _still_ incredibly poor and complex SDLC tooling around managing images.
  - rdtsc 3y ago
    It's like for a few years the whole computing world forgot about security updates and went on a container bender, when woke up one morning realizing what they've done, and then started adding various clunky solutions on top: mutable tags, various tooling to take apart container images and inspect the junk in there, notify and scan for issues etc.
    - pjmlp 3y ago
      You mean like proposing JSON as XML replacement, rebuilding from scratch all the XML tooling including validation, and when almost done with it, replacing JSON with YAML?
      - rdtsc 3y ago
        Exactly! YAML apparently is now terrible as well, but not sure what's next in the pipeline, toml perhaps? I think it would be funny if we reverted back to s-expressions and started the cycle all over.

nurple 3y ago

Containers are a solution to dependency management for sure. In the world of the FHS, the dynamic linker is meant to solve a number of problems, like space saving and security updates, at the OS level by discovering dynamic deps installed in special library paths. One thing I've never liked about FHS is how everything is organized by kind. The interesting thing about how Nix approaches the problem is to replace the concept of FHS almost entirely (only a couple binaries are linked to /) by hijacking PATH, and the linker configs like you mentioned. The biggest difference being that the whole version-pinned dep tree is encoded in a nix package(and in the linker config of the binaries it produces) rather than just the package itself. At some level you could say there is no "dynamic" runtime linking in nix, i.e the linker uses partially specified deps in a discovery phase, all of the link bindings happen at build time. The FHS did attempt to solve the issue of multi-version dependencies with an interesting name and symlink setup, but they are usually still bound by fairly loose version constraints (like major version). Containers are a lot more like nix in this way, where deps are "resolved" at build time by the distro's package manager by virtue of controlling the process' filesystem. This is one major issue with the reproducibility of container builds, the distro package managers are not deterministic, you could run a build back-to-back and get different deps depending on your timing(yes, even between test and build CI steps).

jayd16 3y ago

Containers are powerful because they solve many computing issues, one of which being able to act as a (lower case c) static container for dynamically linked apps as well as cross-language, multi-executable meta-apps. Containers also provide many forms of isolation (network, file system etc.), they provide a modern versioning and distribution scheme, composibility (use another container as a base image). All of these things can, and perhaps should be, done at a the language level as well but containers also work across languages, across linking paradigms, and with existing binaries.

nonameiguess 3y ago

I haven't seen any other response mention it yet, but containers are also heavily used for web-exposed services in part because of address space and port contention. Network namespaces allow you to graft an overlay network onto your physical network in a relatively simple and easy way (not that it's actually easy, but networking never is). Otherwise, sure, nix can rewrite the RPATH in your ELF file to make it pull dynamic libs from the nix store, but what does it do when two processes both want to listen on ports 80 and 443? Possibly, if the Internet ever actually goes pure IPv6, one LAN will have enough addresses to assign one to each process instead of each host. There are, of course, other ways to handle it. People used vhosts predominantly defined in a dedicated web server that was really only a reverse proxy, but now you need nix and nginx. Then you discover you also want resource isolation. Is there a userspace alternative to cGroups? I don't see how there could even in principle be an alternative to PID/UID namespaces and UID/GID submapping. Some things have to happen in the kernel and that means containers of some sort. It doesn't have to be the exact OCI standard that eventually grew out of Docker and eventually Kubernetes, but some kind of container.

adamgordonbell 3y ago
I think what you are saying is true, and my knowledge of networking is pretty slim. But, to play along with my static linking thought-exercise: if you take a process and put it in a network namespace then is it a container? I wouldn't say it is. Container runtimes might have a nice interface for namespacing, but namespacing something doesn't make it a container. I guess my thought experiment is if things are statically linked binaries and you had a way to run them with the control group and namespace settings you wanted, would the packaging aspect of containers add anything? The elites don't want you to know it, but namespaces are just there for the taking. You can grab as many as you want. You can set the memory limit on any process with cgroups, no docker desktop required. :) Anyways, just a thought experiment about how the industry sometimes seems to be going in a circle, in the fashion of the lady who swallowed a fly.

pjmlp 3y ago

That wasn't the deal in HP-UX vaults, Solaris Zones, System 360/MVS virtualization,... The way they are used on GNU/Linux is indeed a solution to GNU/Linux's software distribution issues on a highly fragmented landscape.

bayindirh 3y ago

Disclaimer: I'm not a strong containerization proponent. The good part of containers is you isolating the thing you're running. I'm very against resource waste, but if I can spend 90MB on a container image instead of installing a complete software stack to run a task which is executed weekly and runs for 10 minutes, I'd prefer that. Plus, I can create a virtual network and storage stack around the container(s) if I need to. Case in point: I use imap-backup to backup my e-mail accounts, but it's a ruby application and I need to install the whole stack of things, plus the gems. Instead I containerize it, and keep my system clean. Nix is something different and doesn't solve "many foreign processes not seeing each-other on the same OS" problem. Heck, even Docker doesn't solve all problems, so we have "user level containers" which do not require root access and designed to be run in multitenant systems.

marcosdumay 3y ago
Linux doesn't solve that problem at all. Qubes tries to do that with a hack, that is probably sufficient, but quite complex. I'd say that we don't have good solutions to that problem. But nobody even tried our best candidates, so I'm not completely sure. (For a while, it looked like Android would finally try some. But then Google turned it into a user-hostile anti-privacy OS.)
adamgordonbell 3y ago
> many foreign processes not seeing each-other on the same OS For sure, I was thinking of the packaging nature of containers, not the 'security' nature of containers. The pivotroot part. Though I guess being able to have namespaces does make packaging clearer in certain cases. For a horrible analogy: With actual shipping containers, we don't have each shipping container be a stripped down model of a ship, so that the things in it aren't confused.
- samatman 3y ago
  Agreed, it isn't great. Shipping containers are more like tarballs: a bunch of stuff (files) bundled into one "stuff" (file) so they're easy to ship around and pieces don't get lost in the process. I don't think there is a good physical-world analogy for what containers are doing, though. Maybe takeout including utensils and napkins to "simulate a kitchen" in case you eat it on the go, so you don't have to rely on the "system" forks and napkins? Still kinda rubbish honestly.

Already__Taken 3y ago

dynamic linking to me seems like solving a problem that filesystems should. deduplicating data.

ahepp 3y ago
Deduplication, in the sense of physical space reduction, seems like the least strong argument one could make for dynamic linking these days. The strongest argument I can think of, is enabling system managers (distro maintainers, even end users) to update dependencies. This might be to apply a security patch, enable some kind of tracing for profiling an application, and so on.
pjmlp 3y ago
Plugins have nothing to do with filesystems.
watermelon0 3y ago
In practice, dynamic linking might not be achiving that much in the sense of deduplication: https://drewdevault.com/dynlib.html https://drewdevault.com/dynlib.html
bayindirh 3y ago
As a bonus, shared libraries are deduplicated in RAM, too. Plus you don't need to recompile the universe because of a patch.
brabel 3y ago
That's an interesting way to look at it. If all dynamic libs lived on a read-only location, then the file system could actually only store the libs in one place and the other "copies" would be just symlinks to that... and when the OS loaded such lib, it would automatically know that despite being in different locations, the libs were the same (they're all symlinks to the same place). Is this something that has been attempted before?
- nurple 3y ago
  This is exactly how nix works, except instead of symlinks it actually modifies the binaries at buld-time to point its dynamic library paths to absolute paths in the store which includes a source-derived hash in place of a version.

silverquiet 3y ago

Funny - it looks like you’re being downvoted for asking what I think is a very natural question. It’s one I’ve asked before; have we just created a more elaborate statically-linked executable via containerization? In the end, Docker/OCI seems like the universal Linux package manager. I’m sure I don’t have the full picture since I’m far more ops than dev though.

bayindirh 3y ago
I don't think so. Instead we created an artifact which can live inside an OS, but cannot see and touch to the rest of the OS. CGroups is a deceptively powerful mechanism. You can isolate a process resource wise (X cores, Y amount of memory, Z amount of swap), network wise (a different virtual network adapter with its own IP, bandwidth limits, etc.) and FS wise (running in its own filesystem with devices it can see). It can keep your system tidy by encapsulating elaborate stacks which makes system management painful, allows deterministic operation and image generation if you tag everything with version. Downside is you can do bad things with it like terminating HTTPS with a gateway container and talk HTTP among your backend instead of configuring tools, or writing shoddy software, and getting away with it because it works, or gets automatically restarted when it crashes 6 times a day. I don't run every service as a container, because some services suffocate when they are in a container, but for short running things which needs system-wide changes to function, or test-driving small services before fully committing, it's a good tool to have. However, it's abused with no end, and their popular use leaves a bad taste in your mouth.
- adamgordonbell 3y ago
  You can put any given process is a cgroup. It doesn't have to be a container. If you have a statically linked executable, setup the cgroup for it as you will. No containers needed. You can namespace is as well. No `FROM X` `RUN Y` dockerfile stuff needed.
  - bayindirh 3y ago
    > You can put any given process is a cgroup. It doesn't have to be a container. Yes, I run many programs inside cgroups, but not in containers. > If you have a statically linked executable, setup the cgroup for it as you will. No containers needed. You can namespace is as well. Yes. > No `FROM X` `RUN Y` dockerfile stuff needed. Yes. dockerfile only sets up the chroot in an overlayfs and fires up the "container" using mechanisms present in the kernel already. As I said on another comment, quoting myself: > Docker just made the interface more practical, and built the ecosystem around it.