3 ms·

Fantastic article. I now have something to point people to when they ask "what is wrong with pinning?" or "what is wrong with static linking?" or "why can't you

by tgbugs 6y ago

Fantastic article. I now have something to point people to when they ask "what is wrong with pinning?" or "what is wrong with static linking?" or "why can't you just use pip?" Michał has had to deal with some pretty crazy stuff the last couple of months ... scratch that, years. The recent attempts to empower developers to distribute their own software means that there is now the potential for there to be as many bad security practices as there are pieces of software, because systematic security design that used to be managed by distributions has been pushed down to individual software projects, which have only a tiny view of the issues, and thus repeatedly make locally convenient decisions that are disastrous for everyone else. What do you mean someone is using a project other than ours that shares a dependency? Why would they do that? One other thing to keep in mind is that from a security standpoint the approach that Nix takes is not good. Nix can deal with pinned dependencies on a per-package basis, but if those pinned versions are insecure and the developers don't have a processes for keeping dependencies up to date, then users are sitting ducks. Unfortunately pinning is a symptom of at least three underlying issues. The first is that developers do not properly assess the costs of adding a dependency to a project both in terms of complexity, and in terms of maintenance burden. The second is that many of the dependent libraries make breaking changes without providing space for the old api and the new api to coexist at the same time for a certain period so that developers can transition over (I have been bitten by this with werkzeug and pint). Mutual exclusion of versions due to runtime failure is a nasty problem, and pinning hides that issue. Finally it seems that at least some developers are engaged in the Frog and Toad are Cofounders continuous integration story, but with a twist, rather than deleting failing tests, they pin packages so that they don't have to see the accumulating cost of pulling in additional dependencies. Externalities, externalities everywhere. edit: because it is just too on point https://medium.com/frog-and-toad-are-cofounders/technical-debt-c228faf7bc0f https://medium.com/frog-and-toad-are-cofounders/technical-de...

bscphil 6y ago
> "why can't you just use pip?" IMO, pip is great, but it has exactly two use cases where it's warranted. One is where you need to be your own maintainer, e.g. you've developed a private web server to run your website, and you need to manage its dependency versions precisely. The other is for development purposes: it's really great to be able to use different versions of Python or test against different libraries than your distribution ships. It's not (or shouldn't be) for shipping software to end users. That's the problem being complained about in the essay. This particular way of handling dependencies (use system libraries by default, but allow the user / developer to create an entire virtualized Python installation if they want one) is actually why I think Python handles this better than most other languages, including older ones.
- tgbugs 6y ago
  I agree. The context was missing from my original, which is, "why can't you just use pip to install dependencies in production," which is effectively the answer that Michał once got from a PyPA maintainer when asking about setup.py install.
saurik 6y ago
> I now have something to point people to when they ask "what is wrong with pinning?" or "what is wrong with static linking?" or "why can't you just use pip?" Michał has had to deal with some pretty crazy stuff the last couple of months ... scratch that, years. Literally the only argument in this article against static linking is "it will take an extra couple hours for the distribution to recompile all affected packages and then require the user to download a larger update, meaning time to fix for a security issue will be negligibly longer"... since you seem to believe this article represents the strongest statement of your argument, this has actually moved me further away from where you want me to be ;P. (FWIW, the best argument I could make for dynamic linking involves memory efficiency for shared pages, the disk cache, and maybe the i-cache, not security.)
- jcranmer 6y ago
  > maybe the i-cache This I doubt. On a statically-linked application, all calls to functions are going through a regular function call instruction. When you dynamically link, every call to a function that might cross library boundaries (which on ELF systems defaults to every call that's not to a static function) instead calls into the PLT, which will itself dispatch through to the underlying function call.
  - Lx1oG-AWb6h_ZG0 6y ago
    We had an article on the front page yesterday about Swift’s ABI and how much Apple has invested in dynamic linking: https://news.ycombinator.com/item?id=26205969 https://news.ycombinator.com/item?id=26205969 > It's worth noting that the Swift devs disagree with the Rust and C++ codegen orthodoxy in one major way: they care much more about code sizes (as in the amount of executable code produced). More specifically, they care a lot more about making efficient usage of the cpu's instruction cache, because they believe it's better for system-wide power usage. Apple championing this concern makes a lot of sense, given their suite of battery-powered devices.

by tgbugs 6y ago

bscphil 6y ago

> "why can't you just use pip?" IMO, pip is great, but it has exactly two use cases where it's warranted. One is where you need to be your own maintainer, e.g. you've developed a private web server to run your website, and you need to manage its dependency versions precisely. The other is for development purposes: it's really great to be able to use different versions of Python or test against different libraries than your distribution ships. It's not (or shouldn't be) for shipping software to end users. That's the problem being complained about in the essay. This particular way of handling dependencies (use system libraries by default, but allow the user / developer to create an entire virtualized Python installation if they want one) is actually why I think Python handles this better than most other languages, including older ones.

tgbugs 6y ago
I agree. The context was missing from my original, which is, "why can't you just use pip to install dependencies in production," which is effectively the answer that Michał once got from a PyPA maintainer when asking about setup.py install.

saurik 6y ago

> I now have something to point people to when they ask "what is wrong with pinning?" or "what is wrong with static linking?" or "why can't you just use pip?" Michał has had to deal with some pretty crazy stuff the last couple of months ... scratch that, years. Literally the only argument in this article against static linking is "it will take an extra couple hours for the distribution to recompile all affected packages and then require the user to download a larger update, meaning time to fix for a security issue will be negligibly longer"... since you seem to believe this article represents the strongest statement of your argument, this has actually moved me further away from where you want me to be ;P. (FWIW, the best argument I could make for dynamic linking involves memory efficiency for shared pages, the disk cache, and maybe the i-cache, not security.)

jcranmer 6y ago
> maybe the i-cache This I doubt. On a statically-linked application, all calls to functions are going through a regular function call instruction. When you dynamically link, every call to a function that might cross library boundaries (which on ELF systems defaults to every call that's not to a static function) instead calls into the PLT, which will itself dispatch through to the underlying function call.
- Lx1oG-AWb6h_ZG0 6y ago
  We had an article on the front page yesterday about Swift’s ABI and how much Apple has invested in dynamic linking: https://news.ycombinator.com/item?id=26205969 https://news.ycombinator.com/item?id=26205969 > It's worth noting that the Swift devs disagree with the Rust and C++ codegen orthodoxy in one major way: they care much more about code sizes (as in the amount of executable code produced). More specifically, they care a lot more about making efficient usage of the cpu's instruction cache, because they believe it's better for system-wide power usage. Apple championing this concern makes a lot of sense, given their suite of battery-powered devices.