25 ms·

The modern packager’s security nightmare

320 points by summm 6y ago

7800k 6y ago
Gentoo is secure, as far as you can be secure by building a bunch of code few have time to review on hardware few have time to fully understand running in an insecure world. So, I don’t fault it. However, if you don’t include dependencies and you don’t manage them, which would be the case in modern environments for the majority of users in the world, how is that safe? Answer: it’s not.
setheron 6y ago
Once Nix moves more to fixed output derivations this will be killer! You can easily upgrade dependencies by having the linker resolve to different libraries without needing to rebuild the world while still retaining a sensible memory model.
anotherhue 6y ago
Correct me if I'm wrong (really) but isn't static linking mostly a problem for packagers? The less a package maintainer modifies the application the better IMO. If application A and B rely on dependency D, which turns out to have a vulnerability, fixed in D', Then _why_ do we think it is anyone but A & B's developers responsibility to update to D' and distribute patched versions? If the packager tries to do it, the chances of diverging A and B due to some other incompatibility are too high. Either, you're running well maintained software (OSS or commercial), and get the update in a timely manner, or you're running unmaintained software and don't get the update. Only in the second case does it make sense to patch in a dynamically linked dependency. And I'm sure there's plenty of examples of this, but the real issue is running un-maintained software! I am much happier keeping in close sync with timely releases from Go/Rust projects than I am with Debian et. al's style of freezing the world.
- bombcar 6y ago
  The sad truth is without the package maintaining too many people would be running horribly out of date and insecure software - as there is zero incentive to upgrade a working system.
  - jay_kyburz 6y ago
    I think if the software was labeled "insecure - requires update" people would update. If you don't know that something is insecure then there is zero incentive.
- evmar 6y ago
  The OP is talking about technical details like static linking and language ecosystems, but if you zoom one level out, this comment correctly pins where the underlying problem actually lies: that the distribution maintainers inject themselves into the development process of all the software they ship. When A/B are not well-maintained upstream, distributions can assist by updating its dependency D, but even when A/B are well-maintained upstream, distributions still might monkey with them when D changes, even when that change to D actually breaks the software. To me, the root cause is that the distributions are effectively attempting to participate the software they ship, but do so by getting in the middle, rather than participating upstream. As a former author of some well-maintained upstream software I found their involvement made the software overall worse, but as a user of a distribution I find their maintenance of otherwise unmantained software sometimes helpful. In other words, I think the goal is admirable but the mechanism is wrong, and doing things like adding more dynamic linking only further enable the bad behavior.
  - choeger 6y ago
    I think it would help if distributions could snatch a piece of real estate in upstream software. Something like, "in every project, the debian/ root folder belongs to the debian project and follows their rules". The packaging could then verify this folder and put their patches, build scripts, etc. there. This would help upstream communication a lot, I guess.
    - CJefferson 6y ago
      The problem is there isn't just Debian, there is Debian and arch and Gentoo and nix and guix and freebsd and cygwin (at least). Then, how do I check who should have merge access to all these directories?
      - tgbugs 6y ago
        The problem is actually far worse than that. While the gp has what seems to be a reasonable solution, the issue is that you can't actually know what distribution is going to package your software, or what environment it is going to run in. There are code bases out there that still work decades later without any active maintenance, but if you need maintenance just to be able to build software in a new environment something is wrong. The underlying issue is that there are no standards for how to package software in a multi language environment. If I want to go from a state where (require 'module-name) fails to a state where (require 'module-name) succeeds, there are a potentially infinite number of ways that that could be accomplished, a single software project cannot ever specify all the possible ways for building software. What they can try to do use uniform interfaces and standard patterns for building their software, in a way that delegates dependency management to an external system. It seems that it is hard for software developers to admit that other people know more about how and where their software will be running than they do. Good engineering practice seems to dictate that dependency and environment management should be completely orthogonal to the development of an individual component or individual functionality. The op is 3 stories about what happens when the two are not kept orthogonal. The fundamental problem is that it is often easier for individual projects to make decisions that conflate the individual project with its dependencies (no longer orthogonal). To my knowledge, there is not a universal or well understood set of requirements that could be used to specify what a stable interface between an individual software project and its dependencies looks like. There are a number of candidates, such as gentoo ebuilds, rpm spec files, etc. however I have not seen one that effectively accommodates all of them. Further, there are languages where the implementation (or even design) makes it impossible to keep dependencies and individual projects orthogonal. The end result of non-orthognal systems is more work for everyone, more wasted cpu cycles, and worse security. Distros can't stop people from using languages that conflate the two, but they can tell them that they are on their own, and that the distros can't depend on components written in such languages in the core of the OS. To everyone pushing the rewrite it in Rust meme this should be a wakeup call. The current design decisions in the language and limitations of the implementation make it less secure than C or C++ because swapping out dependencies is bottlenecked by the centralized primary development team, and maintainers and users can't take orthogonal action to fix an issue.
- choeger 6y ago
  The distribution provides support for much longer than upstream does. Also, packaging is actual work. Just because some hipsters in some company consider it cool to release new features every 4 weeks, it does not mean that the volunteers of some linux distribution can keep up with thag. So if you want your distribution to work well, you should focus on workable and transparent interfaces. If you are happy with the support cycles of your upstream and can live with a black box, on the other hand you don't need a distribution in the first place. Besides, the whole "vendor everything, link statically" idea works well only for the leafs of the tree. Guess what Rust would do if llvm was not a readily usable library but a bunch of C++ files directly used inside clang? Thanks to the Rust mode of operation, it is impossible to do with the rust compiler what they did with llvm.
  - anotherhue 6y ago
    > on the other hand you don't need a distribution in the first place. I think you're right about that, I personally want as little distribution as possible. FreeBSD ports or Arch AUR work well for me. The idea of solidifying other people's software into a bundle and then maintaining it (with necessarily limited expertise) seems like a loosing battle.
    - beojan 6y ago
      As someone who uses the AUR a lot, it's really the perfect example of how you just won't update software if it isn't done automatically by the distro package manager, particularly when you use -git packages.
- nwallin 6y ago
  > Correct me if I'm wrong (really) but isn't static linking mostly a problem for packagers? The less a package maintainer modifies the application the better IMO. IMHO that's precisely why dependencies should be unpinned. Let's say application A relies on dependencies B, C, and D, and dependencies B, C, and D depend on dependency E. Let's say dependency E has a critical security vulnerability and needs to be updated today to E'. Let's say you have unpinned versions: The packager for E updates E to E'. The end. Let's say you have pinned versions: The developer for B updates package B to depend on E', the developer for C updates C to depend on E', the developer for D updates D to depend on E', the developer for A updates A to depend on B', C', and D'. The packager for B updates the package for B, the packager for C updates the package for C, the packager for D updates the package for D, and the packager for A updates the package for A. You'll notice that there's a timing issue here. The packager for C cannot move until the developer for C has done the work, the developer for A cannot move until the developer for B, C, and D have done their work, and the packager for A cannot do anything until everyone has completed their work. If, for instance, the developers for C all live in Texas and their power's been out for a few days, and when they get power back they're busy with other stuff for a while, it might take quite some time for C's developers to get an official package posted. But it's that important that A gets updated, because A is a network service with a port open to the internet and E is openssl or whatever. So now what? In a perfect world, all software dependencies would have active, attentive, prompt maintainers, but it tends to not be that way. Lots of critical internet infrastructure packages have a maintainer who's just some random person in Nebraska, and they go on vacation, or lose interest, go to sleep at night, go to little league games on the weekend, some of them have day jobs. If we lived in a world where Apache can't be updated to use the latest dynamic library for openssl because the developer for leftpad is watching a movie and has their phone turned off, that's a very serious problem, and it's a crazy world I would not want to be a sysadmin in. Certainly, maybe the packager for E is gonna be off this week, but a distro's packaging team tends to have a much easier time filling in for a maintainer who's away if the package is loosely coupled with the application's build process. 90% of the time, if a package in Gentoo requires an update, all you need to do is `mv foo-1.2.3.ebuild foo-1.2.4.ebuild`, `repoman manifest` and git commit+push. (I can't speak for other distros.) The system isn't perfect, but IMHO it's much more robust to the unfortunate realities of the ugly, soft underbelly of the world than static linking is.
phkahler 6y ago
>> Why do people pin dependencies? The primary reason is that they don’t want dependency updates to suddenly break their packages for end users, or to have their CI results suddenly broken by third-party changes. Or because we dont want accidental or malicious security vulnerabilities to get automatically incorporated into the software. This stuff works both ways. You dont automatically incorporate fixes, nor new problems.
- lucb1e 6y ago
  The vast, vast majority of updates fix security issues. It's like not vaccinating in case you're one of the million people that has an allergic reaction. Supply chain attacks are rare, not the norm. We hear about such things (and only rarely at that) because it's exceptional enough to make the news.
  - 7sidedmarble 6y ago
    Yeah, but 'pinning' dependencies is useful so that you can choose when you get those changes.
    - lucb1e 6y ago
      Which means extra maintenance work to check for every piece of software that anyone uses whether it uses another library that it needs to be recompiled against and, if it fails, how to use the new version. If there are automatic updates, at least it either works and is more secure, or it breaks automatically and unsafe software stops working. Whether you prefer people to use MSIE6 because "it just works" or whether you prefer old sites that only worked with MSIE6 to break because it's no longer maintained, that's the trade-off you have to choose between. As a security person, I'm obviously biased, I can only advise what I see from a professional perspective. All I was saying above is that automatic updates being considered a security risk is on the same scale of odds as considering vaccines dangerous -- in regular cases, that is: of course the advice is different if you're a special (sensitive) organisation or a special (immunocompromised) person.
  - eecc 6y ago
tgbugs 6y ago
Fantastic article. I now have something to point people to when they ask "what is wrong with pinning?" or "what is wrong with static linking?" or "why can't you just use pip?" Michał has had to deal with some pretty crazy stuff the last couple of months ... scratch that, years. The recent attempts to empower developers to distribute their own software means that there is now the potential for there to be as many bad security practices as there are pieces of software, because systematic security design that used to be managed by distributions has been pushed down to individual software projects, which have only a tiny view of the issues, and thus repeatedly make locally convenient decisions that are disastrous for everyone else. What do you mean someone is using a project other than ours that shares a dependency? Why would they do that? One other thing to keep in mind is that from a security standpoint the approach that Nix takes is not good. Nix can deal with pinned dependencies on a per-package basis, but if those pinned versions are insecure and the developers don't have a processes for keeping dependencies up to date, then users are sitting ducks. Unfortunately pinning is a symptom of at least three underlying issues. The first is that developers do not properly assess the costs of adding a dependency to a project both in terms of complexity, and in terms of maintenance burden. The second is that many of the dependent libraries make breaking changes without providing space for the old api and the new api to coexist at the same time for a certain period so that developers can transition over (I have been bitten by this with werkzeug and pint). Mutual exclusion of versions due to runtime failure is a nasty problem, and pinning hides that issue. Finally it seems that at least some developers are engaged in the Frog and Toad are Cofounders continuous integration story, but with a twist, rather than deleting failing tests, they pin packages so that they don't have to see the accumulating cost of pulling in additional dependencies. Externalities, externalities everywhere. edit: because it is just too on point https://medium.com/frog-and-toad-are-cofounders/technical-debt-c228faf7bc0f https://medium.com/frog-and-toad-are-cofounders/technical-de...
- bscphil 6y ago
  > "why can't you just use pip?" IMO, pip is great, but it has exactly two use cases where it's warranted. One is where you need to be your own maintainer, e.g. you've developed a private web server to run your website, and you need to manage its dependency versions precisely. The other is for development purposes: it's really great to be able to use different versions of Python or test against different libraries than your distribution ships. It's not (or shouldn't be) for shipping software to end users. That's the problem being complained about in the essay. This particular way of handling dependencies (use system libraries by default, but allow the user / developer to create an entire virtualized Python installation if they want one) is actually why I think Python handles this better than most other languages, including older ones.
larusso 6y ago
It’s sounds like pinning dependencies is just done because we developers are a lazy bunch that just want to guard against the rare case of a breaking upstream change. I was burned too often in the past by breaking changes or more often behavior changes in transient dependencies to not put up some defense. But I still don’t pin versions in the main dependency files (cargo.toml, gemset or similar). I just have the generated lockfile and put that under version control. so all developers in the team get the same versions of direct and transient dependencies. I define the dependencies with version ranges. Every package manager works differently here. I like cargo the best since a statement like ‘version = 1.2.1’ means give me a version bigger than ‘1.2.1’ but lower than ‘2.0.0’. One should never add lockfiles for library projects as they need to be open for future updates. I also add dependabot to all repos and let the bot inform me about updates.
mixedCase 6y ago
Yeah well, who's hungry? No really, neither users nor developers care, nor should they. I've been using Linux on the desktop for well over a decade and I'm tired of seeing this plea for everything to behave exactly like C or scripting languages because every distro wants to be its own special snowflake and it would be too hard to adapt. The world has changed and distros have to stop pretending it hasn't. Flatpak, Nix, those are better models that reflect our reality where developers don't want to worry about ten thousand different distros with wildly different downstream configurations producing different kinds of bugs, as well as the desire of users of just getting the darned app. If you're worried about security you must always, first, work with upstream. Upstream should be the party responsible for its users, not you. Upstream not fast enough and you want to go the extra mile? Well then your packaging tool-belt should have support for patching a series of libaries in an automated fashion and rebuilding the dependency tree; and make sure that you return to upstream as soon as possible. If you want your downstream to diverge from upstream because you want to act as a barrier as the old distros do, then you'll have to accept the fact that you're maintaining a fork and not pretend to upstream that you're distributing their work or that your bug reports are compatible. Otherwise, again, just limit yourself on distributing the latest upstream's stable release with the bare minimum patching necessary for getting things to work with the distro's chosen config. With some luck, after the Linux community comes to terms with the fact that the distro model must shift, we can begin to finally share some packaging efforts between all distros and we can leave much of the incompatibility bullshit in the past. Imagine one day everyone just building from shared Nix derivations? Very ironically, it would look a lot like Portage and Gentoo's USE flags but with everyone building off those generic derivations and offering binary caches for them.
- noobermin 6y ago
  It's hard not to read this and not feel like a little bit of it is some maintainers worried about losing control. When I get on irc and say the words "pip" they certainly send me those vibes.
  - loopz 6y ago
    It's just idealism, but no ideology suits everyone and for every context. People forget this.
bscphil 6y ago
Over the last several weeks I was working on an essay about this exact problem, including the connection between static linking and bundling. This one is so well done that I probably won't even publish it. But I'll add this, for people who may not immediately see why this is important. I think that the real danger these new technologies represent is not inherently bad technology, but the possibility of ecosystem damage. The distribution / maintainer / package manager approach has proven to be an extremely reliable way to get trustworthy software. Many of us love it and want to see it stick around. And it's been possible because "upstream" developers in the open source ecosystem have been willing (or forced) to work with distributions to include their software. But this seems to be changing. A highlight from my essay: 'Many software projects are not good citizens in the open source ecosystem. They are working with a model of development and distribution that does not mesh well with how open source software is predominantly produced. ... These [new] languages are now developing their own ecosystems, with their own expectations for how software should be created and distributed, how dependencies should be handled, and what a "good piece of software" looks like. Regardless, an increasing amount of software is being built in these ecosystems, including open source software. There might come a day in which open source is fractured. Two different communities create two very different kinds of software, which run on the same systems, but are created and distributed in very different ways. I begin to worry that the community I care about might not survive. Even if my ecosystem continues on its way, I don't want this split to take place. I think being part of the open source ecosystem is good for software, and I think having all the software you could want available within that ecosystem is good for users. If anything, this essay is a call to those who agree to be more careful about the software they create. Make sure it's something that Debian maintainers could be proud to ship. If you're working on a Rust program, for example, ask questions like "how can I make this program as easy for maintainers to distribute as possible?" If you have the ability to work on projects like Rust or Go, do what you can to give their applications the ability to easily split dependencies and support system provided libraries. Let's try to make sure the software ecosystem we love is around for the next generation.' I'd also strongly recommend the essay "Maintainers Matter" for another take on why this is so important. http://kmkeen.com/maintainers-matter/ http://kmkeen.com/maintainers-matter/ Or read about Debian's recent problems with vendoring: https://lwn.net/Articles/842319/ https://lwn.net/Articles/842319/
choeger 6y ago
Would it be feasible for rust to ship "re-link" scripts? That is, when a cargo dependency gets updated, the distribution can check whether packages code needs to be re-linked? Similarly, when distributed code (say libssl) changes, the rust packages can be re-linked by the distribution on-site?
- charliesome 6y ago
  You can achieve this by recompiling the software. Cargo will not recompile dependencies that have not changed.
IshKebab 6y ago
This article doesn't make a good case against static linking, and the author doesn't seem to understand what vendoring is either: > Bundling (often called vendoring in newspeak) means including the dependencies of your program along with it. No, vendoring means including a copy of the source code of dependencies in your repo. You can bundle dependencies without vendoring them. The only argument presented against static linking is that when a library is updated rebuilding dependants takes longer (and people will have to download bigger updates but I doubt many people care about that). That may be true, but is it really that big of an issue? I somewhat doubt it (for sane distros that don't make users build everything from source themselves anyway). The author clearly doesn't like how people do development these days but hasn't stopped to think why they do it like that.
- UncleEntity 6y ago
  > The only argument presented against static linking is that when a library is updated rebuilding dependants takes longer (and people will have to download bigger updates but I doubt many people care about that). Not really, the argument is that instead of rebuilding one library you now have to rebuild hundreds of applications to see the benefits for whatever the library update was for...assuming upstream even cares enough to bump the version number for the who knows how many libs they are vendoring because they have other stuff to do which is way more important.
  - IshKebab 6y ago
    Right but that can be done automatically for static linking just as easily as it can for dynamic linking. So the only difference is time and downloads. I think you're conflating static linking and bundling and vendoring a bit, like the author is. They're all different things.
posix_me_less 6y ago
Here is the solution to this Debian/Gentoo/<other true free software distribution> packager's dilemma. Packagers realize that their numbers are small and they can't keep up fixing all the modern big software (BS) projects that go against their philosophy. They define the core of the OS that they can keep managing in this traditional way (kernel, basic userland, basic desktop, basic server services, basic libraries, security support for all that). "But people want Firefox, Rust, Kubernetes, modern OS has to provide them". No it doesn't. Let the new complicated juggernaut software be deployed and configured by the users, according to developers' instructions, using whatever modern mechanism they prefer, containers/chroots/light-weight VMs, whatever. Packagers can then forget about the nasty big software and focus on quality of the core OS. Users will have all the newest versions of BS they need from the developers directly. There is no reason to keep fast-pace, ever-changing and very complicated programs/projects in the main OS distribution. It only leads to lots of OS-specific packaging work with mediocre and obsolete results, and that leads to discontent users, developers and packagers.
- pvorb 6y ago
  Exactly. When you want to install one of those fast-paced pieces of software on your system, you're often dissatisfied with the old version coming with your OS anyway.
  - drran 6y ago
    I see no problem to bump versions of a few packages, rebuild them, and install them via system package manager. Are you developer or user? Of course, some software may not work after that, but package manager newer stays in your way, unless you have no idea how to use it.
    - pvorb 6y ago
      I'm a software developer and a user of package managers. A package manager that allows me to only have one version of a software package installed will inevitably get in my way if I need two programs that need different, incompatible versions of that dependency. It happens quite a lot in my experience.
syllogism 6y ago
This will be a somewhat intemperate response, because as a developer of a significant library I found this quite irritating. If you publish a Python library without pinned dependencies, your code is broken. It happens to work today, but there will come a day when the artifact you have published no longer works. It's only a matter of time. The command the user had run before, like "pip install spacy==2.3.5" will no longer work. The user will have to then go to significant trouble to find the set of versions that worked at the time. In short unpinned dependencies mean hopeless bit-rot. It guarantees that your system is a fleeting thing; that you will be unable to today publish an end-to-end set of commands that will work in 2025. This is completely intolerable for practical engineering. In order to fix bugs you may need to go back to prior states of a system and check behaviours. If you can't ever go back and load up a previous version, you'll get into some extremely difficult problems. Of course the people who are doing the work to actually develop these programs refuse to agree to this. No we will not fucking unpin our dependencies. Yes we will tell you to get lost if you ask us to. If you try to do it yourself, I guess we can't stop you, but no we won't volunteer our help. It's maddening to hear people say things like, "Oh if everyone just used semantic versioning this wouldn't be a problem". Of course this cannot work. _Think about it_. There are innumerable ways two pieces of code can be incompatible. You might have a change that alters the time-complexity for niche inputs, making some call time-out that used to succeed. You might introduce a new default keyword argument that throws off a *kwargs. If you call these things "breaking" changes, you will constantly be increasing the major version. But if you increase the major version every release, what's the point of semver! You're not actually conveying any information about whether the changes are "breaking".
- foerbert 6y ago
  I don't have a particularly strong viewpoint on this, but I find it noteworthy that in your example the user themselves is asking for a specific version of the software. You don't seem to be intending for users to ask for simply the latest version and have that work, but a specific one, and you want that specific version to work exactly as it did whenever it was published. I can see some instances in which this expectation is important, and others where it is likely not or else certainly less important than the security implications. For the extremes, I see research using spaCy has a very strong interest in reproducibility and the impact of any security issues would likely be minimal on the whole simply due to the relatively few people likely to run into them. On the other extreme, say some low-level dependency is somehow so compromised simply running the code will end up with the user ransomware'd after just-long-enough that this whole scenario is marginally plausible. Then say spaCy gets incorporated into some other project that goes up the chain a ways and ultimately ends up in LibreOffice. If all of these projects have pinned dependencies, there is now no way to quickly or reasonably create a safe LibreOffice update. It would require a rather large number of people to sequentially update their dependencies, and publish the new version, so that the next project up the chain can do the same. LibreOffice would remain compromised or at best unavailable until the whole chain finished, or else somebody found a way to remove the offending dependency without breaking LibreOffice. I'm not sure how to best reconcile these two competing interests. I think it seems clear that both are important. Even more than that, a particular library might sit on both extremes simultaneously depending on how it is used. The only solution - though a totally unrealistic and terrible one - that comes to mind is to write all code such that all dependencies can be removed without additional work and all dependent features would be automatically disabled. With a standardized listing of these feature-dependency pairs you could even develop more fine-grained workarounds for removal of any feature from any dependency. The sheer scale of possible configurations this would create is utterly horrifying. At any rate, your utter rejection of the article's point seems excessively extreme and even ultimately user-hostile. I can understand your point of view, particularly given the library you develop, however I think you should probably give some more thought to indirect users - ie users of programs that (perhaps ultimately) use spaCy. I don't know that it makes sense to practically change how you do anything, but I don't think the other viewpoint is as utterly wrongheaded as you seem to think.
rosshemsley 6y ago
Whilst I don't totally disagree with many of the points here, I think there's a wider picture to many of these issues. The author is concerned with installing packages on user machines: which are typically very long-lived installs - maybe a user has the same machine with the same dependencies for years. However, for many engineers, (such as myself), a binary may not be used past even a few days from when it was first compiled - e.g. as part of a service in a a quickly continuously integrated system. I might even argue that _most_ software is used in this way. When software is built this way, many of the points in this article are very helpful to keep builds stable and to make deployment fast - and in fact for the case of security, we usually _don't_ want dependencies to auto-update, as we do not want to automatically deploy new code if it has not been audited. Maybe there's a future were OSs become more like this, where binaries are more short lived... maybe not. Although I don't think it's strictly fair to label all of these as "Bad" with a capital B :)
- ampdepolymerase 6y ago
  The way iOS and Fuchsia are dealing with the problem is to completely lockdown the operating system with a tight permissions system. An app can be compromised but the damage is limited. Perhaps it is time for servers to move to a similar model.
  - giantrobot 6y ago
    We should call this newly invented and wholly original concept a "container". The software gets "contained". It just might work. /s
    - eecc 6y ago
      You mean cgroups, or zones don’t you? Docker (was, last time I heard) a security disaster, not generating robust layer hashes, lacking user isolation, and plenty just running as root...
      - giantrobot 6y ago
steveklabnik 6y ago
There is a lot I could say about this article, but I've kinda been in this whole Texas natural disaster situation. But I do think it's worth pointing out one thing: > Rust bundles a huge fork of LLVM It is not super huge, and we try to upstream patches regularly to minimize the fork. We also regularly re-base all current patches on each release of LLVM, so it's less 'fork' and more 'maintain some extra commits to fix bugs.' > and explicitly refuses to support to distributions using the genuine LLVM libraries. We always support the latest LLVM release, and try to maintain compatibilities with older ones as long as is reasonable and possible. IIRC, the last time we raised the base LLVM requirement was in Rust 1.49, at the end of last year. The minimum version it was raised to was LLVM 9, which was released in September of 2019. The current release is 11. Again, because of the lack of the extra patches, you may see miscompilation bugs if you use the stock LLVM. There's pros and cons to every choice here, of course. For more on this, please see https://rustc-dev-guide.rust-lang.org/backend/updating-llvm.html https://rustc-dev-guide.rust-lang.org/backend/updating-llvm.... The Rust project is and has been interested in collaborating on issues to make things easier for folks when there's demand. We've sought feedback from distros in the past and made changes to make things easier! (I also posted a slightly modified version of this to their blog comments as well. EDIT: the author has commented that suggesting Rust's fork is large was in error)
- Daho0n 6y ago
  At a minimum a reasonable time for a package for it to be a good citizen in the system is what is in Debian LTS.
  - steveklabnik 6y ago
    EDIT: you completely changed your comment. My original response is below the line, but I'll also respond to your new one above it. I don't actually understand grammatically what you're asking for or saying in this comment. Rust, Cargo, and a bunch of Rust applications are packaged in Debian today, from Buster onwards. I think what you're saying is that rustc head should be able to be built with the llvm that lives in Debian stable? That is a choice that could be made, I'm sure, but then you're talking about supporting five API-incompatible releases of LLVM. The folks from Debian aren't even asking for that level of compatibility, and it would be quite a bit of work, for unclear benefit. -------------------------------------------- There are lots of demands, from lots of people. Managing a project requires prioritizing different demands of all of your different users. > When will the problem mentioned in the article The article talks about many more things than one single problem. Some range from easy, like the "we already do let you use stock llvm" I mentioned above, and some are difficult and take a long time, like "define a Rust ABI." The needs of packagers need to be balanced with the needs of other folks as well. "define a Rust ABI," specifically, for example, would help some packagers, but it may harm other users. It also may be something that's good to have, but we aren't quite ready for it yet. A willingness to work together to solve issues, and to find solutions that help as many different kinds of folks as possible, is what matters here. We have had folks from other distros pitching in to help make things better for their distros, and I can't imagine we wouldn't accept the help of the OP either.
deleted 6y ago
[deleted]
nomercy400 6y ago
It's a risk calculation: do I want to risk being vulnerable to 0days, or do I want to risk my application not running for any user because a dependency changed its headers/api? As software engineers we want to be in control of as much as possible when running our application, to make it as deterministic as possible. For that we select versions of our dependencies, build it, and then test it, and if our tests pass, we release it. If we would let the OS determine the dependency versions, without testing, what guarantees can we give that our code will work? Do you write tests for future, unknown changes in libraries? Do you write tests with Random, because that's the kind of unpredictability you're introducing?
- regularfry 6y ago
  Keeping the application working in the face of dependency updates is the distro's job, if they're supported in doing it. They won't be pushing dependency updates at random except to channels explicitly marked as unstable (at least, assuming a minimally sane distro). I'm less familiar with Gentoo, but Debian-based distros ought to be safe from that threat (and if anything, you ought to be worries about the reverse problem). So your question becomes: do I want to risk being vulnerable to zero-days?
deleted 6y ago
[deleted]
dataflow 6y ago
Why do only *nix folks pull their hair out about this? On Windows programs bundle their dependencies all the time (sometimes even as shared libraries! but without the independent update benefits) and hardly anybody loses sleep over it. Heck, users actually like the fact that it minimizes friction. Nobody claims it's rock-solid security, but does it need to be? Actually, now that I wrote it above, I think I might have found an answer to my own question: while of course infosec experts will freak out about any vulnerability existing anywhere for even a shred of a nanosecond, in the real world this is really only a big deal for servers, not clients. And I'm guessing this issue affects Linux folks more because servers tend to run on Linux, with security-sensitive software exposed to the open internet all the time. Maybe we need to realize & embrace this trade-off instead of fighting it till the end of time? (I suppose one could even argue servers don't need this either if all the packages are kept up-to-date anyway; I guess that might also be debatable, but it's beside my point.)
- GartzenDeHaes 6y ago
  The unix culture comes from a shared multi-user perspective, whereas the windows point of view tends to be towards a single user desktop. In a shared environment, it's not acceptable for applications to change shared system components. Although this is maybe less important today, the culture still persists.
  - dataflow 6y ago
    > In a shared environment, it's not acceptable for applications to change shared system components. That era is long gone on Windows (literally since XP I think). In fact I've had to do this far more on Linux than on Windows in recent memory... the latest instance ironically being the glibc CLOCK_REALTIME mess on WSL. Right now programs bundle whatever they need, and these go into the app directory (like \Program Files). e.g., I have maybe ~20 sqlite3 DLLs on my system besides the system ones. Your larger point about the culture might still be correct though, I don't know.
    - oblio 6y ago
wheybags 6y ago
Nobody cares how secure my code is if it doesn't work.
- deleted 6y ago
  [deleted]
hctaw 6y ago
The day I stop static linking is the day I can compile on one distro and ship to many without worrying about users reporting loader errors. That day is not today. Until then, I'll keep doing it, because it saves me time and money. I don't really buy that dynamic linking all the things is such a boon to security. But I'll link to this: https://drewdevault.com/dynlib https://drewdevault.com/dynlib > Not including libc, the only libraries which had "critical" or "high" severity vulnerabilities in 2019 which affected over 100 binaries on my system were dbus, gnutls, cairo, libssh2, and curl. 265 binaries were affected by the rest. > The total download cost to upgrade all binaries on my system which were affected by CVEs in 2019 is 3.8 GiB. This is reduced to 1.0 GiB if you eliminate glibc. > It is also unknown if any of these vulnerabilities would have been introduced after the last build date for a given statically linked binary; if so that binary would not need to be updated. Many vulnerabilities are also limited to a specific code path or use-case, and binaries which do not invoke that code path in their dependencies will not be affected. A process to ascertain this information in the wake of a vulnerability could be automated.
- AshamedCaptain 6y ago
  > But I'll link to this: https://drewdevault.com/dynlib https://drewdevault.com/dynlib We already discussed this in HN https://news.ycombinator.com/item?id=23654353 https://news.ycombinator.com/item?id=23654353 . Both data and conclusions are utterly suspect.
- sigjuice 6y ago
  The day I stop static linking is the day I can compile on one distro and ship to many without worrying about users reporting loader errors. That day is not today. Maybe do the trendy thing and link your whole distro together with your app (a.k.a containers)?
  - forrestthewoods 6y ago
    The fact that containers exist and are prevalent is a damning indictment on the Linux model imho. Producing software that can simply launch is so wildly complex that the solution is... to snapshot an entire operating system install to run the application. That’s INSANE. But it’s unfortunately required. https://xkcd.com/1987/ https://xkcd.com/1987/ > Now, for the worst of all — one that combines all the aforementioned issues, and adds even more. Bundling (often called vendoring in newspeak) means including the dependencies of your program along with it. The exact consequences of bundling vary depending on the method used. Let’s consider this statement at face value. The WORST thing a program can do is... include the things the program needs to run? So programs should... NOT include the things they REQUIRE to run? I get it. I understand the argument. But if you take just a half step back it should be clear how utterly broken the whole situation is.
paultopia 6y ago
I feel like this confuses a lot of things by assuming an extremely sophisticated end user. Sure, if you're a double-threat dev/sysadmin using linux, then when some vulnerability gets discovered in some dynamically linked library on your system, you have the capacity to (a) receive information about that fact, and (b) update it. But now suppose you're an ordinary person. You use software. Maybe you even have a windows machine.[1] Which is more likely to actually get a security update to you? (a) You have to update a single piece of software, which you know you've installed, though a recognized distributional channel like an app store or something, and all its dependencies come with it. (b) You have to either learn what a DLL is and learn how to update it and then hope that nothing you rely on breaks in some mysterious way because of some dependency on a dependency on a dependency on a dependency. Or you have to accept a whole operating system update, assuming that the operating system comes with a DLL update---and hence accepting all of the other crap that comes with operating system updates from Microsoft (and Apple), such as bugginess from complex updates, incompatibilities between new versions of operating systems and software (or hardware) that you rely on, new obnoxious security rules that you might not agree to (looking at you, Cupertino), and massive disruptions to your ability to actually use your computer to do your work. No thanks. [1] Maybe this article is specifically targeted against the linux ecosystem? If so, perhaps this issue is ameliorated somewhat, but it still seems to put a fairly substantial burden on end users, that seems to be inconsistent with actually letting non-experts use linux OSes.
- regularfry 6y ago
  Alternatively, realise that the problems in (b) are largely solved for reasonable OSes where the vendor takes responsibility both for automatically getting you security patches and for keeping you working without major disruptions. I'm not sure where you've got the idea that updates are all-or-nothing, but some of us have been living a life you seem to think can't exist for decades at this point.
- deadbunny 6y ago
  That sounds like a good argument for using package managers with automatic security updates. The user doesn't need to be an expert, just reboot when the system tells them to.
ChrisMarshallNY 6y ago
For an OS like Linux, the thing about static linking is 100% spot-on. However, for vetted systems, like app stores, this makes no difference, as the only dylibs allowed to be patched at runtime, are the OS-supplied (ABI) ones. The whole app needs to be rebuilt, re-vetted, and rereleased. Frankly, I would be worried about dylibs, but for a different reason, as more than one hack has been done by hijacking dynamic links. I'm trying to remember how it went, but there was once a common practice for Mac programs to deliberately take advantage of this, in order to add functionality.
Ericson2314 6y ago
This is a moronic by saying everything is equally bad. 1. The linking strategy doesn't matter. Just rebuild everything. The reason Rust and Haskell don't do dynamic linking is pervasive inlining. This is required for many abstractions to work well, and asking optimization boundaries to coincide with software origin boundaries is stupid anyways. The ABI arguement is stupid because replacing all the dynamic libraries with the ABI changes is no worse than rebuilding everything cause static linking. Therefor ignore that part and just think about inlining. 2. Lock files are fine as long as one can in fact bump the minor versions. Yes, we should make tools to enforce minor versions are non-breaking. And it's crazy we didn't have those from day one. But until we do lock files are fine. 3. Vendoring is pure evil.
- Ericson2314 6y ago
  The tl;dr is we always need reproducible and composible builds. You can derive basically everything else from that.
young_unixer 6y ago
Centralized repositories are not a good, general solution to the software distribution problem. First: They don't have all the packages and versions the user could ever need, which means that you'll always have a mixture of software installed through the packagr manager and software installed through tarballs or curl | bash Second: They distort the relationship between application developer and application user. Third: they neglect offline use cases. The only sane, generalizable solution is for the OS to be a platform upon which bundles can be installed. The OS should just define a sane format for said bundles and the user gets their applications directly from the application developer.
kenniskrag 6y ago
Do I need to recompile the shared library and the clients if one of them needs a newer compiler or if the client is provided as binary by the vendor?
zxcvbn4038 6y ago
If your package management is worth it’s salt then you don’t care about static linking, you just update every package that links to that package. IBM’s AIX had the best package management ever IMO, you could roll forward or back and things would just work. Completely deterministic. On the backend whenever a package was updated they would regression test it and everything that depended on it to ensure the update didn’t break anything - when a bug did get through they would fix it and add another regression test so it didn’t happen again. All of that kinda broke when they adopted RPM as a secondary format because the RPMs didn’t make the same guarantees. One of the best features of Golang is being able to cut a single static linked Linux binary that works on Alpine, Red Hat, CentOS, etc. With Go your also not linking to OpenSSL or Libxml, and those two packages are responsible for every emergency roll out I’ve ever had to do. Time and again end result of version pinning is that when forced to update developers are required to advance through multiple major version updates and what should be a patch becomes a rewrite. I have to constantly deal with a bunch of web sites on jquery 2 and the developers boo-hooing every day because jquery 3.5 is completely different and all I can tell them is that jquery 2 is no longer maintained so they need to stop using it for new projects and they need to update or retire their existing projects using it. One of the things I liked about Golang was that it didn’t have any versions to pin so it avoided all of that nonsense altogether. However then they added “modules” and it’s getting harder to not use those, so that killed that magic. Though I did make sure the CI/CD pipelines I use do a go get -u so hopefully that will force the local Go devs to keep their code fresh.
cuillevel3 6y ago
> Static linking, dependency pinning and bundling are three bad practices that have serious impact on the time and effort needed to eliminate vulnerabilities from production systems. I'm astonished how different this perspective is. As a developer I see that software is developed faster nowadays. size, library reuse and functionality are increasing. And distributions are just not able to keep up. I feel like they never did, they just made exceptions for packages that were too important to ignore, like browsers or office suites. Really, it's not the software. It's the distros. Not using libraries is not an option, you don't want devs to write their own crypto. Not pinning dependencies is bad, incompatibilities and security issues could make their way into the code. The test surface also gets bigger and it raises the question which combinations are supported. Update: I think what I want to say is, distributions should accept that they can only provide that level of "stability" for a limited set of applications. The new and shiny stuff will always happen elsewhere.
Kototama 6y ago
> We try hard to unpin the dependencies and test packages with the newest versions of them. However, often we end up discovering that the newer versions of dependencies simply are not compatible with the packages in question. Sadly, upstreams often either ignore reports of these incompatibilities or even are actively hostile to us for not following their pins. I don't get it. They unpin the versions and then are disappointed that the software does not work with untested versions? It would be nice if semantic versioning could always work but it does not. I know Elm, Rust and Nix have some solutions to propose for this problem.
jFriedensreich 6y ago
Well not pinning does not lead to everything that uses the vulnerable dependency to magically rebuild, just to make the builds less deterministic. The solution is not to blame pinning but to use tooling that informs about the vulnerable dependencies.
qwerty456127 6y ago
That's strange the author doesn't mention the madness of replacing classic deb packages with snaps and flatpacks.
X-Istence 6y ago
One of the issues with unbundling everything into its own package really comes to a head when it comes to how Python is packaged on Debian based systems. They split out core components that are built-in to Python so that they may distribute them separately thereby breaking the Python version's default tooling that is supposed to exist, in the name of "not having dev tools installed alongside non dev". Package maintainers have made life miserable for those of us who have to then help people through that mess so that they then may use the software we have written and or maintain. We have to write our quick start guides to tell users how to get a proper working version of Python on their system because the package maintainers deem their way of deploying Python the best way to deploy it. "Your instructions are wrong, `python3 -mvenv venv` doesn't do anything, it just errors out saying I need to install something else, but when I install that package it still doesn't work". Packagers that insist on splitting every dependency then put the onus back on the community to support it, they don't have to deal with it.
bschwindHN 6y ago
You can pry my statically-linked Rust binaries from my cold, dead hands.
1337shadow 6y ago
> Why do people pin dependencies? The primary reason is that they don’t want dependency updates to suddenly break their packages for end users, or to have their CI results suddenly broken by third-party changes. However, all that has another underlying problem — the combination of not being concerned with API stability on upstream part, and not wishing to unnecessarily update working code (that uses deprecated API) on downstream part. Truth is, pinning makes this worse because it sweeps the problem under the carpet, and actively encourages people to develop their code against specific versions of their dependencies rather than against a stable public API. Hyrum’s Law in practice. Exactly one of the problems with pinning your project dependencies, whatever language your project is in. It's better to unpin and continuously integrate upstream changes as early as possible: it's less work this way at the end of the year, and more secure.
caseyavila 6y ago
My goodness, It’s as if [the bundling part of] this post speaks directly to me. I use Gentoo as my daily driver, and school forces us to use Zoom for classes. Like most other proprietary software vendors, Zoom chooses to bundle a great deal of shared objects along with their Linux binary. Gentoo however, chooses to let Zoom use the system libraries instead of those bundled. Through some sort of ABI incompatibility, Zoom’s use of my system’s libraries causes joining a class to get stuck on a “Connecting...” screen. It doesn’t happen every time, but it happens often enough to annoy my teachers, who end up having to let me in multiple times. I don’t really want to run Zoom’s precompiled libraries, and the version numbers of the objects match one to one. But alas, Zoom probably modified the libraries to fit their own applications needs. I can only hope that a future update of Zoom magically solves my problems.

320 points by summm 6y ago

Gentoo is secure, as far as you can be secure by building a bunch of code few have time to review on hardware few have time to fully understand running in an insecure world. So, I don’t fault it. However, if you don’t include dependencies and you don’t manage them, which would be the case in modern environments for the majority of users in the world, how is that safe? Answer: it’s not.

setheron 6y ago

Once Nix moves more to fixed output derivations this will be killer! You can easily upgrade dependencies by having the linker resolve to different libraries without needing to rebuild the world while still retaining a sensible memory model.

anotherhue 6y ago

Correct me if I'm wrong (really) but isn't static linking mostly a problem for packagers? The less a package maintainer modifies the application the better IMO. If application A and B rely on dependency D, which turns out to have a vulnerability, fixed in D', Then _why_ do we think it is anyone but A & B's developers responsibility to update to D' and distribute patched versions? If the packager tries to do it, the chances of diverging A and B due to some other incompatibility are too high. Either, you're running well maintained software (OSS or commercial), and get the update in a timely manner, or you're running unmaintained software and don't get the update. Only in the second case does it make sense to patch in a dynamically linked dependency. And I'm sure there's plenty of examples of this, but the real issue is running un-maintained software! I am much happier keeping in close sync with timely releases from Go/Rust projects than I am with Debian et. al's style of freezing the world.

bombcar 6y ago
The sad truth is without the package maintaining too many people would be running horribly out of date and insecure software - as there is zero incentive to upgrade a working system.
- jay_kyburz 6y ago
  I think if the software was labeled "insecure - requires update" people would update. If you don't know that something is insecure then there is zero incentive.
evmar 6y ago
The OP is talking about technical details like static linking and language ecosystems, but if you zoom one level out, this comment correctly pins where the underlying problem actually lies: that the distribution maintainers inject themselves into the development process of all the software they ship. When A/B are not well-maintained upstream, distributions can assist by updating its dependency D, but even when A/B are well-maintained upstream, distributions still might monkey with them when D changes, even when that change to D actually breaks the software. To me, the root cause is that the distributions are effectively attempting to participate the software they ship, but do so by getting in the middle, rather than participating upstream. As a former author of some well-maintained upstream software I found their involvement made the software overall worse, but as a user of a distribution I find their maintenance of otherwise unmantained software sometimes helpful. In other words, I think the goal is admirable but the mechanism is wrong, and doing things like adding more dynamic linking only further enable the bad behavior.
- choeger 6y ago
  I think it would help if distributions could snatch a piece of real estate in upstream software. Something like, "in every project, the debian/ root folder belongs to the debian project and follows their rules". The packaging could then verify this folder and put their patches, build scripts, etc. there. This would help upstream communication a lot, I guess.
  - CJefferson 6y ago
    The problem is there isn't just Debian, there is Debian and arch and Gentoo and nix and guix and freebsd and cygwin (at least). Then, how do I check who should have merge access to all these directories?
    - tgbugs 6y ago
      The problem is actually far worse than that. While the gp has what seems to be a reasonable solution, the issue is that you can't actually know what distribution is going to package your software, or what environment it is going to run in. There are code bases out there that still work decades later without any active maintenance, but if you need maintenance just to be able to build software in a new environment something is wrong. The underlying issue is that there are no standards for how to package software in a multi language environment. If I want to go from a state where (require 'module-name) fails to a state where (require 'module-name) succeeds, there are a potentially infinite number of ways that that could be accomplished, a single software project cannot ever specify all the possible ways for building software. What they can try to do use uniform interfaces and standard patterns for building their software, in a way that delegates dependency management to an external system. It seems that it is hard for software developers to admit that other people know more about how and where their software will be running than they do. Good engineering practice seems to dictate that dependency and environment management should be completely orthogonal to the development of an individual component or individual functionality. The op is 3 stories about what happens when the two are not kept orthogonal. The fundamental problem is that it is often easier for individual projects to make decisions that conflate the individual project with its dependencies (no longer orthogonal). To my knowledge, there is not a universal or well understood set of requirements that could be used to specify what a stable interface between an individual software project and its dependencies looks like. There are a number of candidates, such as gentoo ebuilds, rpm spec files, etc. however I have not seen one that effectively accommodates all of them. Further, there are languages where the implementation (or even design) makes it impossible to keep dependencies and individual projects orthogonal. The end result of non-orthognal systems is more work for everyone, more wasted cpu cycles, and worse security. Distros can't stop people from using languages that conflate the two, but they can tell them that they are on their own, and that the distros can't depend on components written in such languages in the core of the OS. To everyone pushing the rewrite it in Rust meme this should be a wakeup call. The current design decisions in the language and limitations of the implementation make it less secure than C or C++ because swapping out dependencies is bottlenecked by the centralized primary development team, and maintainers and users can't take orthogonal action to fix an issue.
choeger 6y ago
The distribution provides support for much longer than upstream does. Also, packaging is actual work. Just because some hipsters in some company consider it cool to release new features every 4 weeks, it does not mean that the volunteers of some linux distribution can keep up with thag. So if you want your distribution to work well, you should focus on workable and transparent interfaces. If you are happy with the support cycles of your upstream and can live with a black box, on the other hand you don't need a distribution in the first place. Besides, the whole "vendor everything, link statically" idea works well only for the leafs of the tree. Guess what Rust would do if llvm was not a readily usable library but a bunch of C++ files directly used inside clang? Thanks to the Rust mode of operation, it is impossible to do with the rust compiler what they did with llvm.
- anotherhue 6y ago
  > on the other hand you don't need a distribution in the first place. I think you're right about that, I personally want as little distribution as possible. FreeBSD ports or Arch AUR work well for me. The idea of solidifying other people's software into a bundle and then maintaining it (with necessarily limited expertise) seems like a loosing battle.
  - beojan 6y ago
    As someone who uses the AUR a lot, it's really the perfect example of how you just won't update software if it isn't done automatically by the distro package manager, particularly when you use -git packages.
nwallin 6y ago
> Correct me if I'm wrong (really) but isn't static linking mostly a problem for packagers? The less a package maintainer modifies the application the better IMO. IMHO that's precisely why dependencies should be unpinned. Let's say application A relies on dependencies B, C, and D, and dependencies B, C, and D depend on dependency E. Let's say dependency E has a critical security vulnerability and needs to be updated today to E'. Let's say you have unpinned versions: The packager for E updates E to E'. The end. Let's say you have pinned versions: The developer for B updates package B to depend on E', the developer for C updates C to depend on E', the developer for D updates D to depend on E', the developer for A updates A to depend on B', C', and D'. The packager for B updates the package for B, the packager for C updates the package for C, the packager for D updates the package for D, and the packager for A updates the package for A. You'll notice that there's a timing issue here. The packager for C cannot move until the developer for C has done the work, the developer for A cannot move until the developer for B, C, and D have done their work, and the packager for A cannot do anything until everyone has completed their work. If, for instance, the developers for C all live in Texas and their power's been out for a few days, and when they get power back they're busy with other stuff for a while, it might take quite some time for C's developers to get an official package posted. But it's that important that A gets updated, because A is a network service with a port open to the internet and E is openssl or whatever. So now what? In a perfect world, all software dependencies would have active, attentive, prompt maintainers, but it tends to not be that way. Lots of critical internet infrastructure packages have a maintainer who's just some random person in Nebraska, and they go on vacation, or lose interest, go to sleep at night, go to little league games on the weekend, some of them have day jobs. If we lived in a world where Apache can't be updated to use the latest dynamic library for openssl because the developer for leftpad is watching a movie and has their phone turned off, that's a very serious problem, and it's a crazy world I would not want to be a sysadmin in. Certainly, maybe the packager for E is gonna be off this week, but a distro's packaging team tends to have a much easier time filling in for a maintainer who's away if the package is loosely coupled with the application's build process. 90% of the time, if a package in Gentoo requires an update, all you need to do is `mv foo-1.2.3.ebuild foo-1.2.4.ebuild`, `repoman manifest` and git commit+push. (I can't speak for other distros.) The system isn't perfect, but IMHO it's much more robust to the unfortunate realities of the ugly, soft underbelly of the world than static linking is.

phkahler 6y ago

>> Why do people pin dependencies? The primary reason is that they don’t want dependency updates to suddenly break their packages for end users, or to have their CI results suddenly broken by third-party changes. Or because we dont want accidental or malicious security vulnerabilities to get automatically incorporated into the software. This stuff works both ways. You dont automatically incorporate fixes, nor new problems.

lucb1e 6y ago
The vast, vast majority of updates fix security issues. It's like not vaccinating in case you're one of the million people that has an allergic reaction. Supply chain attacks are rare, not the norm. We hear about such things (and only rarely at that) because it's exceptional enough to make the news.
- 7sidedmarble 6y ago
  Yeah, but 'pinning' dependencies is useful so that you can choose when you get those changes.
  - lucb1e 6y ago
    Which means extra maintenance work to check for every piece of software that anyone uses whether it uses another library that it needs to be recompiled against and, if it fails, how to use the new version. If there are automatic updates, at least it either works and is more secure, or it breaks automatically and unsafe software stops working. Whether you prefer people to use MSIE6 because "it just works" or whether you prefer old sites that only worked with MSIE6 to break because it's no longer maintained, that's the trade-off you have to choose between. As a security person, I'm obviously biased, I can only advise what I see from a professional perspective. All I was saying above is that automatic updates being considered a security risk is on the same scale of odds as considering vaccines dangerous -- in regular cases, that is: of course the advice is different if you're a special (sensitive) organisation or a special (immunocompromised) person.
- eecc 6y ago

tgbugs 6y ago

Fantastic article. I now have something to point people to when they ask "what is wrong with pinning?" or "what is wrong with static linking?" or "why can't you just use pip?" Michał has had to deal with some pretty crazy stuff the last couple of months ... scratch that, years. The recent attempts to empower developers to distribute their own software means that there is now the potential for there to be as many bad security practices as there are pieces of software, because systematic security design that used to be managed by distributions has been pushed down to individual software projects, which have only a tiny view of the issues, and thus repeatedly make locally convenient decisions that are disastrous for everyone else. What do you mean someone is using a project other than ours that shares a dependency? Why would they do that? One other thing to keep in mind is that from a security standpoint the approach that Nix takes is not good. Nix can deal with pinned dependencies on a per-package basis, but if those pinned versions are insecure and the developers don't have a processes for keeping dependencies up to date, then users are sitting ducks. Unfortunately pinning is a symptom of at least three underlying issues. The first is that developers do not properly assess the costs of adding a dependency to a project both in terms of complexity, and in terms of maintenance burden. The second is that many of the dependent libraries make breaking changes without providing space for the old api and the new api to coexist at the same time for a certain period so that developers can transition over (I have been bitten by this with werkzeug and pint). Mutual exclusion of versions due to runtime failure is a nasty problem, and pinning hides that issue. Finally it seems that at least some developers are engaged in the Frog and Toad are Cofounders continuous integration story, but with a twist, rather than deleting failing tests, they pin packages so that they don't have to see the accumulating cost of pulling in additional dependencies. Externalities, externalities everywhere. edit: because it is just too on point https://medium.com/frog-and-toad-are-cofounders/technical-debt-c228faf7bc0f https://medium.com/frog-and-toad-are-cofounders/technical-de...

bscphil 6y ago
> "why can't you just use pip?" IMO, pip is great, but it has exactly two use cases where it's warranted. One is where you need to be your own maintainer, e.g. you've developed a private web server to run your website, and you need to manage its dependency versions precisely. The other is for development purposes: it's really great to be able to use different versions of Python or test against different libraries than your distribution ships. It's not (or shouldn't be) for shipping software to end users. That's the problem being complained about in the essay. This particular way of handling dependencies (use system libraries by default, but allow the user / developer to create an entire virtualized Python installation if they want one) is actually why I think Python handles this better than most other languages, including older ones.

larusso 6y ago

It’s sounds like pinning dependencies is just done because we developers are a lazy bunch that just want to guard against the rare case of a breaking upstream change. I was burned too often in the past by breaking changes or more often behavior changes in transient dependencies to not put up some defense. But I still don’t pin versions in the main dependency files (cargo.toml, gemset or similar). I just have the generated lockfile and put that under version control. so all developers in the team get the same versions of direct and transient dependencies. I define the dependencies with version ranges. Every package manager works differently here. I like cargo the best since a statement like ‘version = 1.2.1’ means give me a version bigger than ‘1.2.1’ but lower than ‘2.0.0’. One should never add lockfiles for library projects as they need to be open for future updates. I also add dependabot to all repos and let the bot inform me about updates.

mixedCase 6y ago

Yeah well, who's hungry? No really, neither users nor developers care, nor should they. I've been using Linux on the desktop for well over a decade and I'm tired of seeing this plea for everything to behave exactly like C or scripting languages because every distro wants to be its own special snowflake and it would be too hard to adapt. The world has changed and distros have to stop pretending it hasn't. Flatpak, Nix, those are better models that reflect our reality where developers don't want to worry about ten thousand different distros with wildly different downstream configurations producing different kinds of bugs, as well as the desire of users of just getting the darned app. If you're worried about security you must always, first, work with upstream. Upstream should be the party responsible for its users, not you. Upstream not fast enough and you want to go the extra mile? Well then your packaging tool-belt should have support for patching a series of libaries in an automated fashion and rebuilding the dependency tree; and make sure that you return to upstream as soon as possible. If you want your downstream to diverge from upstream because you want to act as a barrier as the old distros do, then you'll have to accept the fact that you're maintaining a fork and not pretend to upstream that you're distributing their work or that your bug reports are compatible. Otherwise, again, just limit yourself on distributing the latest upstream's stable release with the bare minimum patching necessary for getting things to work with the distro's chosen config. With some luck, after the Linux community comes to terms with the fact that the distro model must shift, we can begin to finally share some packaging efforts between all distros and we can leave much of the incompatibility bullshit in the past. Imagine one day everyone just building from shared Nix derivations? Very ironically, it would look a lot like Portage and Gentoo's USE flags but with everyone building off those generic derivations and offering binary caches for them.

noobermin 6y ago
It's hard not to read this and not feel like a little bit of it is some maintainers worried about losing control. When I get on irc and say the words "pip" they certainly send me those vibes.
- loopz 6y ago
  It's just idealism, but no ideology suits everyone and for every context. People forget this.

bscphil 6y ago

Over the last several weeks I was working on an essay about this exact problem, including the connection between static linking and bundling. This one is so well done that I probably won't even publish it. But I'll add this, for people who may not immediately see why this is important. I think that the real danger these new technologies represent is not inherently bad technology, but the possibility of ecosystem damage. The distribution / maintainer / package manager approach has proven to be an extremely reliable way to get trustworthy software. Many of us love it and want to see it stick around. And it's been possible because "upstream" developers in the open source ecosystem have been willing (or forced) to work with distributions to include their software. But this seems to be changing. A highlight from my essay: 'Many software projects are not good citizens in the open source ecosystem. They are working with a model of development and distribution that does not mesh well with how open source software is predominantly produced. ... These [new] languages are now developing their own ecosystems, with their own expectations for how software should be created and distributed, how dependencies should be handled, and what a "good piece of software" looks like. Regardless, an increasing amount of software is being built in these ecosystems, including open source software. There might come a day in which open source is fractured. Two different communities create two very different kinds of software, which run on the same systems, but are created and distributed in very different ways. I begin to worry that the community I care about might not survive. Even if my ecosystem continues on its way, I don't want this split to take place. I think being part of the open source ecosystem is good for software, and I think having all the software you could want available within that ecosystem is good for users. If anything, this essay is a call to those who agree to be more careful about the software they create. Make sure it's something that Debian maintainers could be proud to ship. If you're working on a Rust program, for example, ask questions like "how can I make this program as easy for maintainers to distribute as possible?" If you have the ability to work on projects like Rust or Go, do what you can to give their applications the ability to easily split dependencies and support system provided libraries. Let's try to make sure the software ecosystem we love is around for the next generation.' I'd also strongly recommend the essay "Maintainers Matter" for another take on why this is so important. http://kmkeen.com/maintainers-matter/ http://kmkeen.com/maintainers-matter/ Or read about Debian's recent problems with vendoring: https://lwn.net/Articles/842319/ https://lwn.net/Articles/842319/

choeger 6y ago

Would it be feasible for rust to ship "re-link" scripts? That is, when a cargo dependency gets updated, the distribution can check whether packages code needs to be re-linked? Similarly, when distributed code (say libssl) changes, the rust packages can be re-linked by the distribution on-site?

charliesome 6y ago
You can achieve this by recompiling the software. Cargo will not recompile dependencies that have not changed.

IshKebab 6y ago

This article doesn't make a good case against static linking, and the author doesn't seem to understand what vendoring is either: > Bundling (often called vendoring in newspeak) means including the dependencies of your program along with it. No, vendoring means including a copy of the source code of dependencies in your repo. You can bundle dependencies without vendoring them. The only argument presented against static linking is that when a library is updated rebuilding dependants takes longer (and people will have to download bigger updates but I doubt many people care about that). That may be true, but is it really that big of an issue? I somewhat doubt it (for sane distros that don't make users build everything from source themselves anyway). The author clearly doesn't like how people do development these days but hasn't stopped to think why they do it like that.

UncleEntity 6y ago
> The only argument presented against static linking is that when a library is updated rebuilding dependants takes longer (and people will have to download bigger updates but I doubt many people care about that). Not really, the argument is that instead of rebuilding one library you now have to rebuild hundreds of applications to see the benefits for whatever the library update was for...assuming upstream even cares enough to bump the version number for the who knows how many libs they are vendoring because they have other stuff to do which is way more important.
- IshKebab 6y ago
  Right but that can be done automatically for static linking just as easily as it can for dynamic linking. So the only difference is time and downloads. I think you're conflating static linking and bundling and vendoring a bit, like the author is. They're all different things.

posix_me_less 6y ago

Here is the solution to this Debian/Gentoo/<other true free software distribution> packager's dilemma. Packagers realize that their numbers are small and they can't keep up fixing all the modern big software (BS) projects that go against their philosophy. They define the core of the OS that they can keep managing in this traditional way (kernel, basic userland, basic desktop, basic server services, basic libraries, security support for all that). "But people want Firefox, Rust, Kubernetes, modern OS has to provide them". No it doesn't. Let the new complicated juggernaut software be deployed and configured by the users, according to developers' instructions, using whatever modern mechanism they prefer, containers/chroots/light-weight VMs, whatever. Packagers can then forget about the nasty big software and focus on quality of the core OS. Users will have all the newest versions of BS they need from the developers directly. There is no reason to keep fast-pace, ever-changing and very complicated programs/projects in the main OS distribution. It only leads to lots of OS-specific packaging work with mediocre and obsolete results, and that leads to discontent users, developers and packagers.

pvorb 6y ago
Exactly. When you want to install one of those fast-paced pieces of software on your system, you're often dissatisfied with the old version coming with your OS anyway.
- drran 6y ago
  I see no problem to bump versions of a few packages, rebuild them, and install them via system package manager. Are you developer or user? Of course, some software may not work after that, but package manager newer stays in your way, unless you have no idea how to use it.
  - pvorb 6y ago
    I'm a software developer and a user of package managers. A package manager that allows me to only have one version of a software package installed will inevitably get in my way if I need two programs that need different, incompatible versions of that dependency. It happens quite a lot in my experience.

syllogism 6y ago

This will be a somewhat intemperate response, because as a developer of a significant library I found this quite irritating. If you publish a Python library without pinned dependencies, your code is broken. It happens to work today, but there will come a day when the artifact you have published no longer works. It's only a matter of time. The command the user had run before, like "pip install spacy==2.3.5" will no longer work. The user will have to then go to significant trouble to find the set of versions that worked at the time. In short unpinned dependencies mean hopeless bit-rot. It guarantees that your system is a fleeting thing; that you will be unable to today publish an end-to-end set of commands that will work in 2025. This is completely intolerable for practical engineering. In order to fix bugs you may need to go back to prior states of a system and check behaviours. If you can't ever go back and load up a previous version, you'll get into some extremely difficult problems. Of course the people who are doing the work to actually develop these programs refuse to agree to this. No we will not fucking unpin our dependencies. Yes we will tell you to get lost if you ask us to. If you try to do it yourself, I guess we can't stop you, but no we won't volunteer our help. It's maddening to hear people say things like, "Oh if everyone just used semantic versioning this wouldn't be a problem". Of course this cannot work. _Think about it_. There are innumerable ways two pieces of code can be incompatible. You might have a change that alters the time-complexity for niche inputs, making some call time-out that used to succeed. You might introduce a new default keyword argument that throws off a *kwargs. If you call these things "breaking" changes, you will constantly be increasing the major version. But if you increase the major version every release, what's the point of semver! You're not actually conveying any information about whether the changes are "breaking".

foerbert 6y ago
I don't have a particularly strong viewpoint on this, but I find it noteworthy that in your example the user themselves is asking for a specific version of the software. You don't seem to be intending for users to ask for simply the latest version and have that work, but a specific one, and you want that specific version to work exactly as it did whenever it was published. I can see some instances in which this expectation is important, and others where it is likely not or else certainly less important than the security implications. For the extremes, I see research using spaCy has a very strong interest in reproducibility and the impact of any security issues would likely be minimal on the whole simply due to the relatively few people likely to run into them. On the other extreme, say some low-level dependency is somehow so compromised simply running the code will end up with the user ransomware'd after just-long-enough that this whole scenario is marginally plausible. Then say spaCy gets incorporated into some other project that goes up the chain a ways and ultimately ends up in LibreOffice. If all of these projects have pinned dependencies, there is now no way to quickly or reasonably create a safe LibreOffice update. It would require a rather large number of people to sequentially update their dependencies, and publish the new version, so that the next project up the chain can do the same. LibreOffice would remain compromised or at best unavailable until the whole chain finished, or else somebody found a way to remove the offending dependency without breaking LibreOffice. I'm not sure how to best reconcile these two competing interests. I think it seems clear that both are important. Even more than that, a particular library might sit on both extremes simultaneously depending on how it is used. The only solution - though a totally unrealistic and terrible one - that comes to mind is to write all code such that all dependencies can be removed without additional work and all dependent features would be automatically disabled. With a standardized listing of these feature-dependency pairs you could even develop more fine-grained workarounds for removal of any feature from any dependency. The sheer scale of possible configurations this would create is utterly horrifying. At any rate, your utter rejection of the article's point seems excessively extreme and even ultimately user-hostile. I can understand your point of view, particularly given the library you develop, however I think you should probably give some more thought to indirect users - ie users of programs that (perhaps ultimately) use spaCy. I don't know that it makes sense to practically change how you do anything, but I don't think the other viewpoint is as utterly wrongheaded as you seem to think.

rosshemsley 6y ago

Whilst I don't totally disagree with many of the points here, I think there's a wider picture to many of these issues. The author is concerned with installing packages on user machines: which are typically very long-lived installs - maybe a user has the same machine with the same dependencies for years. However, for many engineers, (such as myself), a binary may not be used past even a few days from when it was first compiled - e.g. as part of a service in a a quickly continuously integrated system. I might even argue that _most_ software is used in this way. When software is built this way, many of the points in this article are very helpful to keep builds stable and to make deployment fast - and in fact for the case of security, we usually _don't_ want dependencies to auto-update, as we do not want to automatically deploy new code if it has not been audited. Maybe there's a future were OSs become more like this, where binaries are more short lived... maybe not. Although I don't think it's strictly fair to label all of these as "Bad" with a capital B :)

ampdepolymerase 6y ago
The way iOS and Fuchsia are dealing with the problem is to completely lockdown the operating system with a tight permissions system. An app can be compromised but the damage is limited. Perhaps it is time for servers to move to a similar model.
- giantrobot 6y ago
  We should call this newly invented and wholly original concept a "container". The software gets "contained". It just might work. /s
  - eecc 6y ago
    You mean cgroups, or zones don’t you? Docker (was, last time I heard) a security disaster, not generating robust layer hashes, lacking user isolation, and plenty just running as root...
    - giantrobot 6y ago

steveklabnik 6y ago

There is a lot I could say about this article, but I've kinda been in this whole Texas natural disaster situation. But I do think it's worth pointing out one thing: > Rust bundles a huge fork of LLVM It is not super huge, and we try to upstream patches regularly to minimize the fork. We also regularly re-base all current patches on each release of LLVM, so it's less 'fork' and more 'maintain some extra commits to fix bugs.' > and explicitly refuses to support to distributions using the genuine LLVM libraries. We always support the latest LLVM release, and try to maintain compatibilities with older ones as long as is reasonable and possible. IIRC, the last time we raised the base LLVM requirement was in Rust 1.49, at the end of last year. The minimum version it was raised to was LLVM 9, which was released in September of 2019. The current release is 11. Again, because of the lack of the extra patches, you may see miscompilation bugs if you use the stock LLVM. There's pros and cons to every choice here, of course. For more on this, please see https://rustc-dev-guide.rust-lang.org/backend/updating-llvm.html https://rustc-dev-guide.rust-lang.org/backend/updating-llvm.... The Rust project is and has been interested in collaborating on issues to make things easier for folks when there's demand. We've sought feedback from distros in the past and made changes to make things easier! (I also posted a slightly modified version of this to their blog comments as well. EDIT: the author has commented that suggesting Rust's fork is large was in error)

Daho0n 6y ago
At a minimum a reasonable time for a package for it to be a good citizen in the system is what is in Debian LTS.
- steveklabnik 6y ago
  EDIT: you completely changed your comment. My original response is below the line, but I'll also respond to your new one above it. I don't actually understand grammatically what you're asking for or saying in this comment. Rust, Cargo, and a bunch of Rust applications are packaged in Debian today, from Buster onwards. I think what you're saying is that rustc head should be able to be built with the llvm that lives in Debian stable? That is a choice that could be made, I'm sure, but then you're talking about supporting five API-incompatible releases of LLVM. The folks from Debian aren't even asking for that level of compatibility, and it would be quite a bit of work, for unclear benefit. -------------------------------------------- There are lots of demands, from lots of people. Managing a project requires prioritizing different demands of all of your different users. > When will the problem mentioned in the article The article talks about many more things than one single problem. Some range from easy, like the "we already do let you use stock llvm" I mentioned above, and some are difficult and take a long time, like "define a Rust ABI." The needs of packagers need to be balanced with the needs of other folks as well. "define a Rust ABI," specifically, for example, would help some packagers, but it may harm other users. It also may be something that's good to have, but we aren't quite ready for it yet. A willingness to work together to solve issues, and to find solutions that help as many different kinds of folks as possible, is what matters here. We have had folks from other distros pitching in to help make things better for their distros, and I can't imagine we wouldn't accept the help of the OP either.

deleted 6y ago

[deleted]

nomercy400 6y ago

It's a risk calculation: do I want to risk being vulnerable to 0days, or do I want to risk my application not running for any user because a dependency changed its headers/api? As software engineers we want to be in control of as much as possible when running our application, to make it as deterministic as possible. For that we select versions of our dependencies, build it, and then test it, and if our tests pass, we release it. If we would let the OS determine the dependency versions, without testing, what guarantees can we give that our code will work? Do you write tests for future, unknown changes in libraries? Do you write tests with Random, because that's the kind of unpredictability you're introducing?

regularfry 6y ago
Keeping the application working in the face of dependency updates is the distro's job, if they're supported in doing it. They won't be pushing dependency updates at random except to channels explicitly marked as unstable (at least, assuming a minimally sane distro). I'm less familiar with Gentoo, but Debian-based distros ought to be safe from that threat (and if anything, you ought to be worries about the reverse problem). So your question becomes: do I want to risk being vulnerable to zero-days?

deleted 6y ago

[deleted]

dataflow 6y ago

Why do only *nix folks pull their hair out about this? On Windows programs bundle their dependencies all the time (sometimes even as shared libraries! but without the independent update benefits) and hardly anybody loses sleep over it. Heck, users actually like the fact that it minimizes friction. Nobody claims it's rock-solid security, but does it need to be? Actually, now that I wrote it above, I think I might have found an answer to my own question: while of course infosec experts will freak out about any vulnerability existing anywhere for even a shred of a nanosecond, in the real world this is really only a big deal for servers, not clients. And I'm guessing this issue affects Linux folks more because servers tend to run on Linux, with security-sensitive software exposed to the open internet all the time. Maybe we need to realize & embrace this trade-off instead of fighting it till the end of time? (I suppose one could even argue servers don't need this either if all the packages are kept up-to-date anyway; I guess that might also be debatable, but it's beside my point.)

GartzenDeHaes 6y ago
The unix culture comes from a shared multi-user perspective, whereas the windows point of view tends to be towards a single user desktop. In a shared environment, it's not acceptable for applications to change shared system components. Although this is maybe less important today, the culture still persists.
- dataflow 6y ago
  > In a shared environment, it's not acceptable for applications to change shared system components. That era is long gone on Windows (literally since XP I think). In fact I've had to do this far more on Linux than on Windows in recent memory... the latest instance ironically being the glibc CLOCK_REALTIME mess on WSL. Right now programs bundle whatever they need, and these go into the app directory (like \Program Files). e.g., I have maybe ~20 sqlite3 DLLs on my system besides the system ones. Your larger point about the culture might still be correct though, I don't know.
  - oblio 6y ago

wheybags 6y ago

Nobody cares how secure my code is if it doesn't work.

deleted 6y ago
[deleted]

hctaw 6y ago

The day I stop static linking is the day I can compile on one distro and ship to many without worrying about users reporting loader errors. That day is not today. Until then, I'll keep doing it, because it saves me time and money. I don't really buy that dynamic linking all the things is such a boon to security. But I'll link to this: https://drewdevault.com/dynlib https://drewdevault.com/dynlib > Not including libc, the only libraries which had "critical" or "high" severity vulnerabilities in 2019 which affected over 100 binaries on my system were dbus, gnutls, cairo, libssh2, and curl. 265 binaries were affected by the rest. > The total download cost to upgrade all binaries on my system which were affected by CVEs in 2019 is 3.8 GiB. This is reduced to 1.0 GiB if you eliminate glibc. > It is also unknown if any of these vulnerabilities would have been introduced after the last build date for a given statically linked binary; if so that binary would not need to be updated. Many vulnerabilities are also limited to a specific code path or use-case, and binaries which do not invoke that code path in their dependencies will not be affected. A process to ascertain this information in the wake of a vulnerability could be automated.

AshamedCaptain 6y ago
> But I'll link to this: https://drewdevault.com/dynlib https://drewdevault.com/dynlib We already discussed this in HN https://news.ycombinator.com/item?id=23654353 https://news.ycombinator.com/item?id=23654353 . Both data and conclusions are utterly suspect.
sigjuice 6y ago
The day I stop static linking is the day I can compile on one distro and ship to many without worrying about users reporting loader errors. That day is not today. Maybe do the trendy thing and link your whole distro together with your app (a.k.a containers)?
- forrestthewoods 6y ago
  The fact that containers exist and are prevalent is a damning indictment on the Linux model imho. Producing software that can simply launch is so wildly complex that the solution is... to snapshot an entire operating system install to run the application. That’s INSANE. But it’s unfortunately required. https://xkcd.com/1987/ https://xkcd.com/1987/ > Now, for the worst of all — one that combines all the aforementioned issues, and adds even more. Bundling (often called vendoring in newspeak) means including the dependencies of your program along with it. The exact consequences of bundling vary depending on the method used. Let’s consider this statement at face value. The WORST thing a program can do is... include the things the program needs to run? So programs should... NOT include the things they REQUIRE to run? I get it. I understand the argument. But if you take just a half step back it should be clear how utterly broken the whole situation is.

paultopia 6y ago

I feel like this confuses a lot of things by assuming an extremely sophisticated end user. Sure, if you're a double-threat dev/sysadmin using linux, then when some vulnerability gets discovered in some dynamically linked library on your system, you have the capacity to (a) receive information about that fact, and (b) update it. But now suppose you're an ordinary person. You use software. Maybe you even have a windows machine.[1] Which is more likely to actually get a security update to you? (a) You have to update a single piece of software, which you know you've installed, though a recognized distributional channel like an app store or something, and all its dependencies come with it. (b) You have to either learn what a DLL is and learn how to update it and then hope that nothing you rely on breaks in some mysterious way because of some dependency on a dependency on a dependency on a dependency. Or you have to accept a whole operating system update, assuming that the operating system comes with a DLL update---and hence accepting all of the other crap that comes with operating system updates from Microsoft (and Apple), such as bugginess from complex updates, incompatibilities between new versions of operating systems and software (or hardware) that you rely on, new obnoxious security rules that you might not agree to (looking at you, Cupertino), and massive disruptions to your ability to actually use your computer to do your work. No thanks. [1] Maybe this article is specifically targeted against the linux ecosystem? If so, perhaps this issue is ameliorated somewhat, but it still seems to put a fairly substantial burden on end users, that seems to be inconsistent with actually letting non-experts use linux OSes.

regularfry 6y ago
Alternatively, realise that the problems in (b) are largely solved for reasonable OSes where the vendor takes responsibility both for automatically getting you security patches and for keeping you working without major disruptions. I'm not sure where you've got the idea that updates are all-or-nothing, but some of us have been living a life you seem to think can't exist for decades at this point.
deadbunny 6y ago
That sounds like a good argument for using package managers with automatic security updates. The user doesn't need to be an expert, just reboot when the system tells them to.

ChrisMarshallNY 6y ago

For an OS like Linux, the thing about static linking is 100% spot-on. However, for vetted systems, like app stores, this makes no difference, as the only dylibs allowed to be patched at runtime, are the OS-supplied (ABI) ones. The whole app needs to be rebuilt, re-vetted, and rereleased. Frankly, I would be worried about dylibs, but for a different reason, as more than one hack has been done by hijacking dynamic links. I'm trying to remember how it went, but there was once a common practice for Mac programs to deliberately take advantage of this, in order to add functionality.

Ericson2314 6y ago

This is a moronic by saying everything is equally bad. 1. The linking strategy doesn't matter. Just rebuild everything. The reason Rust and Haskell don't do dynamic linking is pervasive inlining. This is required for many abstractions to work well, and asking optimization boundaries to coincide with software origin boundaries is stupid anyways. The ABI arguement is stupid because replacing all the dynamic libraries with the ABI changes is no worse than rebuilding everything cause static linking. Therefor ignore that part and just think about inlining. 2. Lock files are fine as long as one can in fact bump the minor versions. Yes, we should make tools to enforce minor versions are non-breaking. And it's crazy we didn't have those from day one. But until we do lock files are fine. 3. Vendoring is pure evil.

Ericson2314 6y ago
The tl;dr is we always need reproducible and composible builds. You can derive basically everything else from that.

young_unixer 6y ago

Centralized repositories are not a good, general solution to the software distribution problem. First: They don't have all the packages and versions the user could ever need, which means that you'll always have a mixture of software installed through the packagr manager and software installed through tarballs or curl | bash Second: They distort the relationship between application developer and application user. Third: they neglect offline use cases. The only sane, generalizable solution is for the OS to be a platform upon which bundles can be installed. The OS should just define a sane format for said bundles and the user gets their applications directly from the application developer.

kenniskrag 6y ago

Do I need to recompile the shared library and the clients if one of them needs a newer compiler or if the client is provided as binary by the vendor?

zxcvbn4038 6y ago

If your package management is worth it’s salt then you don’t care about static linking, you just update every package that links to that package. IBM’s AIX had the best package management ever IMO, you could roll forward or back and things would just work. Completely deterministic. On the backend whenever a package was updated they would regression test it and everything that depended on it to ensure the update didn’t break anything - when a bug did get through they would fix it and add another regression test so it didn’t happen again. All of that kinda broke when they adopted RPM as a secondary format because the RPMs didn’t make the same guarantees. One of the best features of Golang is being able to cut a single static linked Linux binary that works on Alpine, Red Hat, CentOS, etc. With Go your also not linking to OpenSSL or Libxml, and those two packages are responsible for every emergency roll out I’ve ever had to do. Time and again end result of version pinning is that when forced to update developers are required to advance through multiple major version updates and what should be a patch becomes a rewrite. I have to constantly deal with a bunch of web sites on jquery 2 and the developers boo-hooing every day because jquery 3.5 is completely different and all I can tell them is that jquery 2 is no longer maintained so they need to stop using it for new projects and they need to update or retire their existing projects using it. One of the things I liked about Golang was that it didn’t have any versions to pin so it avoided all of that nonsense altogether. However then they added “modules” and it’s getting harder to not use those, so that killed that magic. Though I did make sure the CI/CD pipelines I use do a go get -u so hopefully that will force the local Go devs to keep their code fresh.

cuillevel3 6y ago

> Static linking, dependency pinning and bundling are three bad practices that have serious impact on the time and effort needed to eliminate vulnerabilities from production systems. I'm astonished how different this perspective is. As a developer I see that software is developed faster nowadays. size, library reuse and functionality are increasing. And distributions are just not able to keep up. I feel like they never did, they just made exceptions for packages that were too important to ignore, like browsers or office suites. Really, it's not the software. It's the distros. Not using libraries is not an option, you don't want devs to write their own crypto. Not pinning dependencies is bad, incompatibilities and security issues could make their way into the code. The test surface also gets bigger and it raises the question which combinations are supported. Update: I think what I want to say is, distributions should accept that they can only provide that level of "stability" for a limited set of applications. The new and shiny stuff will always happen elsewhere.

Kototama 6y ago

> We try hard to unpin the dependencies and test packages with the newest versions of them. However, often we end up discovering that the newer versions of dependencies simply are not compatible with the packages in question. Sadly, upstreams often either ignore reports of these incompatibilities or even are actively hostile to us for not following their pins. I don't get it. They unpin the versions and then are disappointed that the software does not work with untested versions? It would be nice if semantic versioning could always work but it does not. I know Elm, Rust and Nix have some solutions to propose for this problem.

jFriedensreich 6y ago

Well not pinning does not lead to everything that uses the vulnerable dependency to magically rebuild, just to make the builds less deterministic. The solution is not to blame pinning but to use tooling that informs about the vulnerable dependencies.

qwerty456127 6y ago

That's strange the author doesn't mention the madness of replacing classic deb packages with snaps and flatpacks.

X-Istence 6y ago

One of the issues with unbundling everything into its own package really comes to a head when it comes to how Python is packaged on Debian based systems. They split out core components that are built-in to Python so that they may distribute them separately thereby breaking the Python version's default tooling that is supposed to exist, in the name of "not having dev tools installed alongside non dev". Package maintainers have made life miserable for those of us who have to then help people through that mess so that they then may use the software we have written and or maintain. We have to write our quick start guides to tell users how to get a proper working version of Python on their system because the package maintainers deem their way of deploying Python the best way to deploy it. "Your instructions are wrong, `python3 -mvenv venv` doesn't do anything, it just errors out saying I need to install something else, but when I install that package it still doesn't work". Packagers that insist on splitting every dependency then put the onus back on the community to support it, they don't have to deal with it.

bschwindHN 6y ago

You can pry my statically-linked Rust binaries from my cold, dead hands.

1337shadow 6y ago

> Why do people pin dependencies? The primary reason is that they don’t want dependency updates to suddenly break their packages for end users, or to have their CI results suddenly broken by third-party changes. However, all that has another underlying problem — the combination of not being concerned with API stability on upstream part, and not wishing to unnecessarily update working code (that uses deprecated API) on downstream part. Truth is, pinning makes this worse because it sweeps the problem under the carpet, and actively encourages people to develop their code against specific versions of their dependencies rather than against a stable public API. Hyrum’s Law in practice. Exactly one of the problems with pinning your project dependencies, whatever language your project is in. It's better to unpin and continuously integrate upstream changes as early as possible: it's less work this way at the end of the year, and more secure.

caseyavila 6y ago

My goodness, It’s as if [the bundling part of] this post speaks directly to me. I use Gentoo as my daily driver, and school forces us to use Zoom for classes. Like most other proprietary software vendors, Zoom chooses to bundle a great deal of shared objects along with their Linux binary. Gentoo however, chooses to let Zoom use the system libraries instead of those bundled. Through some sort of ABI incompatibility, Zoom’s use of my system’s libraries causes joining a class to get stuck on a “Connecting...” screen. It doesn’t happen every time, but it happens often enough to annoy my teachers, who end up having to let me in multiple times. I don’t really want to run Zoom’s precompiled libraries, and the version numbers of the objects match one to one. But alas, Zoom probably modified the libraries to fit their own applications needs. I can only hope that a future update of Zoom magically solves my problems.