5 ms·

I guess one can work around the lack of dynamic linking in a language like Go on the package manager and/or distro level (though some established distros and pa

by c0l0 5y ago

I guess one can work around the lack of dynamic linking in a language like Go on the package manager and/or distro level (though some established distros and package management ecosystems, which I very much hope will not go away, will have it harder to adapt than others), but I really would like to make everyone think about whether we should... Imagine a Debian archive (that is, a few tens of thousands of packages with source, build recipies and binaries for multiple architectures available) with only static linking on the table. You got a few hundreds to thousands of those packages installed on typical, real-world Debian machines. Then, a central component like zlib or OpenSSL gets an important patch. This triggers hundreds, if not tens of thousands (transitive library .so dependencies are a thing after all, and you want to have all your supported arches covered) of rebuilds across the archive. All users will have to re-download and re-install all resulting upgraded packages they have installed. With increased sharing of source code (i.e., when following "don't build your own crypto!" and other generally useful mantras by re-using existing codebases) and only static linking on the table, this kind of problems gets really, really bad really quickly for large-ish software archives. Kinda like with log4j in the Java ecosystem, where each app is used and event expected to dragging all its runtime dependencies along on its own. In the dylib world we are living in right now however, users get a single updated package, and upon the next restart of an application linking against it, they're immunized against the bug. The reduction of churn is nigh-unfathomably massive. If your systems generally only run a single application, and there's a dedicated ops team at the top (or bottom - it's a matter of perspective, I guess :)) of a monorepo world with each dependency fully vendored in, and an "intelligent" build and deployment system atop it all that will take care of gracefully upgrading everything everywhere upon such routine patching, I can see how statically linking everything reduces the amount of potential pain points upon deploying stuff. For the significant portion that makes up the rest of the world, they generally don't have such luxuries, and will be a lot worse off. Especially established and wonderful Free Software communities like Debian or Fedora. I think that would be a huge sacrifice to make.

gravypod 5y ago
> I guess one can work around the lack of dynamic linking in a language like Go on the package manager and/or distro level (though some established distros and package management ecosystems, which I very much hope will not go away, will have it harder to adapt than others) Package managers already have ways to do this since no programs are 100% dynamicly or statically linked. Even Golang has dynamic linking (glibc recently). > but I really would like to make everyone think about whether we should... Why not? This has the major benefits of making distributed rebuilds, caching of builds, etc much easier. Nix and Bazel have been very successful with this. > Imagine a Debian archive (that is, a few tens of thousands of packages with source, build recipies and binaries for multiple architectures available) with only static linking on the table. You got a few hundreds to thousands of those packages installed on typical, real-world Debian machines. Then, a central component like zlib or OpenSSL gets an important patch. This triggers hundreds, if not tens of thousands (transitive library .so dependencies are a thing after all, and you want to have all your supported arches covered) of rebuilds across the archive. All users will have to re-download and re-install all resulting upgraded packages they have installed. The story isn't as bad as you might think. I'd wager that most programs do not dynamically link anything other than glibc: https://drewdevault.com/dynlib https://drewdevault.com/dynlib > With increased sharing of source code ... and only static linking on the table, this kind of problems gets really, really bad really quickly for large-ish software archives. Kinda like with log4j in the Java ecosystem, where each app is used and event expected to dragging all its runtime dependencies along on its own. With LTOs it's possible to remove the 99% of the code that isn't used from these libraries. It's not as bad as one might think and, if you define your libraries in smaller scopes, you can include only a few .so files. > In the dylib world we are living in right now however, users get a single updated package, and upon the next restart of an application linking against it, they're immunized against the bug This is the positive case but the unfortunate negative is that a majority of the time the program breaks. To save 30MB to 100MB of network traffic we introduce the possibility of an breaking all bins. An overwhelming majority of my binaries use less than 10% of the symbols of their dynamic libraries. Also, if things are statically linked, we can run automated testing on binaries to catch errors before deploying package updates. Maybe a security fix breaks your favorite app? Useful info to know. > of a monorepo world with each dependency fully vendored in, and an "intelligent" build and deployment system atop it all that will take care of gracefully upgrading everything That's sort of the description of what OS distros do IMO. > For the significant portion that makes up the rest of the world, they generally don't have such luxuries, and will be a lot worse off. Especially established and wonderful Free Software communities like Debian or Fedora. I think that would be a huge sacrifice to make. IMO having a system like what Nix has to do hermetic builds would improve debian and other OSs as they could build, test, and reproduce builds more easily. This is important for security.

by c0l0 5y ago

gravypod 5y ago

> I guess one can work around the lack of dynamic linking in a language like Go on the package manager and/or distro level (though some established distros and package management ecosystems, which I very much hope will not go away, will have it harder to adapt than others) Package managers already have ways to do this since no programs are 100% dynamicly or statically linked. Even Golang has dynamic linking (glibc recently). > but I really would like to make everyone think about whether we should... Why not? This has the major benefits of making distributed rebuilds, caching of builds, etc much easier. Nix and Bazel have been very successful with this. > Imagine a Debian archive (that is, a few tens of thousands of packages with source, build recipies and binaries for multiple architectures available) with only static linking on the table. You got a few hundreds to thousands of those packages installed on typical, real-world Debian machines. Then, a central component like zlib or OpenSSL gets an important patch. This triggers hundreds, if not tens of thousands (transitive library .so dependencies are a thing after all, and you want to have all your supported arches covered) of rebuilds across the archive. All users will have to re-download and re-install all resulting upgraded packages they have installed. The story isn't as bad as you might think. I'd wager that most programs do not dynamically link anything other than glibc: https://drewdevault.com/dynlib https://drewdevault.com/dynlib > With increased sharing of source code ... and only static linking on the table, this kind of problems gets really, really bad really quickly for large-ish software archives. Kinda like with log4j in the Java ecosystem, where each app is used and event expected to dragging all its runtime dependencies along on its own. With LTOs it's possible to remove the 99% of the code that isn't used from these libraries. It's not as bad as one might think and, if you define your libraries in smaller scopes, you can include only a few .so files. > In the dylib world we are living in right now however, users get a single updated package, and upon the next restart of an application linking against it, they're immunized against the bug This is the positive case but the unfortunate negative is that a majority of the time the program breaks. To save 30MB to 100MB of network traffic we introduce the possibility of an breaking all bins. An overwhelming majority of my binaries use less than 10% of the symbols of their dynamic libraries. Also, if things are statically linked, we can run automated testing on binaries to catch errors before deploying package updates. Maybe a security fix breaks your favorite app? Useful info to know. > of a monorepo world with each dependency fully vendored in, and an "intelligent" build and deployment system atop it all that will take care of gracefully upgrading everything That's sort of the description of what OS distros do IMO. > For the significant portion that makes up the rest of the world, they generally don't have such luxuries, and will be a lot worse off. Especially established and wonderful Free Software communities like Debian or Fedora. I think that would be a huge sacrifice to make. IMO having a system like what Nix has to do hermetic builds would improve debian and other OSs as they could build, test, and reproduce builds more easily. This is important for security.