8 ms·

Python libraries shipped by distributions are so old that this mechanism is mostly useless for python development. This also applies to many others programming

by elurg 5y ago

Python libraries shipped by distributions are so old that this mechanism is mostly useless for python development. This also applies to many others programming languages which have their own packaging systems. While python packaging is indeed messy the needs of traditional linux distributions are by far the least important. Python packaging needs to better serve the needs of python developers, not those of sysadmins. Having a "linux distro" interpose itself between you and your libraries is a fundamentally broken system not worth fixing. All decisions related to dependency choices fundamentally belongs with upstream.

dgan 5y ago
Exactly agreed on that. I don't understand what value distribution is providing by repackaging python libs, they re always way too old to be usable, and they re global while I work on many projects, with their own incompatible requirements. Maybe I am dumb, but I exclusively use virtualenv and pip..
- avidphantasm 5y ago
  I agree with this for pure Python libraries. However, as soon as you get into things that bind to C libraries (Numpy, GDAL, etc.) it quickly becomes much easier to use the package from your distro.
- rlpb 5y ago
  > I don't understand what value distribution is providing by repackaging python libs... I want to easily and safely use some app my distribution ships. I want to receive security updates automatically for all such apps. I don't care what language it's written in or what its dependencies are. These app packages provided by the distribution have dependencies that are also packaged by the distribution so that dependency resolution works. Since the point of a distribution is that it can run apps, the value is that a distribution works at all.
  - dgan 5y ago
    Valid remark. I had "development side" point of view, not "usage"
turminal 5y ago
Language specific package managers are the antithesis of package management. There is no "management" in "install multiple versions of stuff in this 10 levels deep dependency tree". Upstreams can pick their dependencies but at the same time cannot control what packages depend on them and make it into your app and neither can you. That's a job for your distro.
- pfranz 5y ago
  What happens if your project needs an update, but it will hose the OS? Do you expect someone working in Python in Windows who wants to distribute a datetime package should package for RPM and APT? I used to think OS package managers should be the end-all be-all, but the use cases for OS package managers are very different from language runtimes. While different Linux distros were fighting between themselves, they completely ignored use cases for projects and language runtimes. Sadly, the result is a mess for everyone. I think people need to figure out what approach works best for them. I'm of the opinion that any core technology to your business needs to be decoupled from the OS. It makes OS updates too messy and you're tethered to whatever the OS supports. My field uses a lot of Python and every company figured out quickly they need to run their own binaries in addition to packages. At one company, they packaged up custom RPMs. It wouldn't be a problem to package up and distribute Python libraries. Others had their own package system (no OS or runtime fit their needs). It seems like most people use something like virtualenv. Regrettably, this means there's no easy answer for people new to Python and the right choice will probably change as you grow. But I really think the answer is Python should come up with something that works for Python and let OSes do their thing.
  - turminal 5y ago
    > What happens if your project needs an update, but it will hose the OS? Whay do you mean by "needs an update"? > Do you expect someone working in Python in Windows who wants to distribute a datetime package should package for RPM and APT? Absolutely not. I expect a serious user of that library on an apt based system to package it and submit the package to their distro.
samwillis 5y ago
The fundamental problem is the difference between Curated and Uncurated packaging systems: - Linux dictros have curated packaging, often outdated but stable. - PyPi, NPM, and others are Uncurated, anyone can publish, could be unstable, could be insecure. It's down to the developer to decide which route they want to take, and at the moment most want to move quickly with the latest tools. To do that you have to go the uncurated route. It only becomes an issue for a developer if their software is published by a Linux package management system, but 99.9% of developers will never have that.
- rvdginste 5y ago
  > The fundamental problem is the difference between Curated and Uncurated packaging systems Exactly this! It is expected from linux distros that they have curated packaging. I think that is good and I really expect it to stay that way. Whether you want curated or uncurated packages depends on the use case. As the user of a program, I definitely want curated packages. As the developer of a program, I want to specify myself which version of a dependency I want to develop against and I don't want to be hindered by the linux distro in doing that. I do think that developers in that context are not always supported that well in linux distros. And on the other hand, I do think that tools supported by the programming language can assist in that scenario. (for example installing multiple versions of the compiler and runtime in the users home directory and being able to easily switch between those versions on a project basis...)
Denvercoder9 5y ago
> All decisions related to dependency choices fundamentally belongs with upstream. No. As a user I want dependency management (and all of software distribution, to be honest) to be handled by the party that's best able to keep things working while at the same time keeping them secure. Linux distributions have a much, much better track record at that than most upstreams.
- Gigachad 5y ago
  I really doubt that the python libraries packaged by Debian are any more secure or stable than the latest release of those libraries. At best they just limit to breaking updates to once every few years when they update them. It’s essentially like version locking packages except some random Debian maintainer decides when it’s time to update.
  - Denvercoder9 5y ago
    > I really doubt that the python libraries packaged by Debian are any more secure or stable than the latest release of those libraries. They are more stable because I can keep using the same version for two years, and I'm not being pushed to the latest version that has (intentional or unintentional) breaking changes every two months. Yes, there might be a bug or two in there that have since been fixed, but I very much prefer the failure I know over unexpected failures. They are secure because Debian (and distros like it) backport security fixes to their packages. You can argue about whether they do a good enough job keeping up with vulnerabilities, but at least I know that once I install the update from Debian, my machine is secure, and I don't have to wait for the upstream authors of all software on my machine to release updates that upgrade their dependency. > It’s essentially like version locking packages except some random Debian maintainer decides when it’s time to update. Yes, but version locking isn't my problem. The crucial difference is that distros pick a version and support those for years, while upstreams usually force you to use the latest version all the time to get security support. With distros _I_ get to decide when I upgrade, and the reduced frequency is a nice bonus. Having a single entity for all software on the system is also valuable, as there's just one tool to learn and one place to check for updates.
  - 5y ago
cjfd 5y ago
Perhaps the distributions would be more inclined to include up-to-date version if the standard in the python community was not to break everything all the time. There are distributions that keep up-to-date, though, e.g., archlinux. Serving the needs of python developers vs. sysadmins is a false dichotomy. Python developers develop on a system that they need to admin. One great thing about linux is that everything on a system can be kept up-to-date using just one software tool (a package manager). You are not going to convince me in a million years that this is a bad idea. Now, for things that are needed in development you sometimes need different versions, in particular if you happen to have an OS that ships very old versions. For that purpose there should be some sort of tool and not a gazillion tools that are all incompatible and behave in slightly (or not so slightly) different ways. 'python -m venv' is different from 'virtualenv'? WTF? Also, if your distribution is up-to-date (e.g., archlinux) and you are pinning to older version there is also a problem. As the article puts it 'pin their dependencies to 10 versions and 6 vulnerabilities ago'. And if you actually want to maintain your software in the future you, at some point, have to go to the new version anyway.
- BiteCode_dev 5y ago
  Sure, so my lib works on mac, windows, centos and debian. Tell me how I'm supposed to define my lib python deps ? How am I suppose to package the lib ? Distribute it ? Deal with os versions differences? Allow isolation for my user projects ?
  - 1337shadow 5y ago
    I'm sorry, doesn't pip already works there? Otherwise, there's pyinstaller which is great but it requires an entrypoint so it won't do with a pure lib that doesn't expose any script at all.
    - BiteCode_dev 5y ago
      Yes that's the point. We won't use distro packages because only regular python packages allow this. Edit : arf, sorry, I answered the wrong comment, I actually agree with my parent.
rollcat 5y ago
> Python packaging needs to better serve the needs of python developers, not those of sysadmins. Please substitute s/sysadmins/users/, and realise your developers are users too. I've been doing Python for almost 15 years; and I'm getting really fed up with some things. Packaging is a mess. Distribution is a mess (for servers/IoT - Docker saves the day; for desktop - I feel like giving up). Managing the installations is a mess; upgrades can be impossible - I'm hard-stuck on Python3.6 on one project! I find myself rewriting many smaller tools in Go or Rust, just because I can upgrade the toolchain at any time, and/or ship a static binary. But Rust has a very high barrier to entry, and Go tends to be simplistic. I'd fully jump ship today, but Python has just too much momentum behind it.
- ShamelessC 5y ago
  I agree to some extent, but I believe they were mostly trying to convey that much of the work is done by volunteers who are largely only motivated/have the resources to test a small subset of potential deployment scenarios. Those contributions are still valuable and contributors themselves don't owe you anything. The best way to fix this issue would be to get involved (or switch languages as you suggested, that's fine too).
drran 5y ago
> Python packaging needs to better serve the needs of python developers, not those of sysadmins. Use Fedora, or Arch, or another developer-oriented distro.
goodpoint 5y ago
> Python libraries shipped by distributions are so old that this mechanism is mostly useless for python development. Absolutely not. Most distributions ship versions that are released before the distro release freeze. The whole point of using a distro is to have a reliable and trusted development and production platform.
regularfry 5y ago
Yes, exactly this. Only I view it from the other way round: to try to do development with distro-supplied language packages is a category error. Distro-supplied interpreters and their associated libraries are there for the applications supplied and supported by the distribution. Unless you are developing something to be part of the distro, they are not for you. Do not let the distro get between you and your libraries. Supply your own.
- Pokepokalypse 5y ago
  >Unless you are developing something to be part of the distro, they are not for you. . . . forgetting the original purpose and mindset behind Linux in the first place. From a 1990's point of view; it was always intended to be a hobbyist OS; and in most cases, one had to compile all the binaries and kernel one's self. There was no such thing, really, as a "user" or "admin" - everyone was considered, and expected to be a developer.
  - regularfry 5y ago
    Yes, and distributions arose to solve the problem that plagued early Linux: how to have a set of applications and libraries with consistent and mutually compatible versions of everything. If you're compiling everything yourself, building it so it all works once is an achievement. Keeping on top of as many moving targets as there are binaries in the system so that everything keeps working over time is not practical once the number of moving parts gets high enough. Distros solve a real problem, but the trade-off is that some parts of the system must exist to serve itself.

by elurg 5y ago

dgan 5y ago

Exactly agreed on that. I don't understand what value distribution is providing by repackaging python libs, they re always way too old to be usable, and they re global while I work on many projects, with their own incompatible requirements. Maybe I am dumb, but I exclusively use virtualenv and pip..

avidphantasm 5y ago
I agree with this for pure Python libraries. However, as soon as you get into things that bind to C libraries (Numpy, GDAL, etc.) it quickly becomes much easier to use the package from your distro.
rlpb 5y ago
> I don't understand what value distribution is providing by repackaging python libs... I want to easily and safely use some app my distribution ships. I want to receive security updates automatically for all such apps. I don't care what language it's written in or what its dependencies are. These app packages provided by the distribution have dependencies that are also packaged by the distribution so that dependency resolution works. Since the point of a distribution is that it can run apps, the value is that a distribution works at all.
- dgan 5y ago
  Valid remark. I had "development side" point of view, not "usage"

turminal 5y ago

Language specific package managers are the antithesis of package management. There is no "management" in "install multiple versions of stuff in this 10 levels deep dependency tree". Upstreams can pick their dependencies but at the same time cannot control what packages depend on them and make it into your app and neither can you. That's a job for your distro.

pfranz 5y ago
What happens if your project needs an update, but it will hose the OS? Do you expect someone working in Python in Windows who wants to distribute a datetime package should package for RPM and APT? I used to think OS package managers should be the end-all be-all, but the use cases for OS package managers are very different from language runtimes. While different Linux distros were fighting between themselves, they completely ignored use cases for projects and language runtimes. Sadly, the result is a mess for everyone. I think people need to figure out what approach works best for them. I'm of the opinion that any core technology to your business needs to be decoupled from the OS. It makes OS updates too messy and you're tethered to whatever the OS supports. My field uses a lot of Python and every company figured out quickly they need to run their own binaries in addition to packages. At one company, they packaged up custom RPMs. It wouldn't be a problem to package up and distribute Python libraries. Others had their own package system (no OS or runtime fit their needs). It seems like most people use something like virtualenv. Regrettably, this means there's no easy answer for people new to Python and the right choice will probably change as you grow. But I really think the answer is Python should come up with something that works for Python and let OSes do their thing.
- turminal 5y ago
  > What happens if your project needs an update, but it will hose the OS? Whay do you mean by "needs an update"? > Do you expect someone working in Python in Windows who wants to distribute a datetime package should package for RPM and APT? Absolutely not. I expect a serious user of that library on an apt based system to package it and submit the package to their distro.

samwillis 5y ago

The fundamental problem is the difference between Curated and Uncurated packaging systems: - Linux dictros have curated packaging, often outdated but stable. - PyPi, NPM, and others are Uncurated, anyone can publish, could be unstable, could be insecure. It's down to the developer to decide which route they want to take, and at the moment most want to move quickly with the latest tools. To do that you have to go the uncurated route. It only becomes an issue for a developer if their software is published by a Linux package management system, but 99.9% of developers will never have that.

rvdginste 5y ago
> The fundamental problem is the difference between Curated and Uncurated packaging systems Exactly this! It is expected from linux distros that they have curated packaging. I think that is good and I really expect it to stay that way. Whether you want curated or uncurated packages depends on the use case. As the user of a program, I definitely want curated packages. As the developer of a program, I want to specify myself which version of a dependency I want to develop against and I don't want to be hindered by the linux distro in doing that. I do think that developers in that context are not always supported that well in linux distros. And on the other hand, I do think that tools supported by the programming language can assist in that scenario. (for example installing multiple versions of the compiler and runtime in the users home directory and being able to easily switch between those versions on a project basis...)

Denvercoder9 5y ago

> All decisions related to dependency choices fundamentally belongs with upstream. No. As a user I want dependency management (and all of software distribution, to be honest) to be handled by the party that's best able to keep things working while at the same time keeping them secure. Linux distributions have a much, much better track record at that than most upstreams.

Gigachad 5y ago
I really doubt that the python libraries packaged by Debian are any more secure or stable than the latest release of those libraries. At best they just limit to breaking updates to once every few years when they update them. It’s essentially like version locking packages except some random Debian maintainer decides when it’s time to update.
- Denvercoder9 5y ago
  > I really doubt that the python libraries packaged by Debian are any more secure or stable than the latest release of those libraries. They are more stable because I can keep using the same version for two years, and I'm not being pushed to the latest version that has (intentional or unintentional) breaking changes every two months. Yes, there might be a bug or two in there that have since been fixed, but I very much prefer the failure I know over unexpected failures. They are secure because Debian (and distros like it) backport security fixes to their packages. You can argue about whether they do a good enough job keeping up with vulnerabilities, but at least I know that once I install the update from Debian, my machine is secure, and I don't have to wait for the upstream authors of all software on my machine to release updates that upgrade their dependency. > It’s essentially like version locking packages except some random Debian maintainer decides when it’s time to update. Yes, but version locking isn't my problem. The crucial difference is that distros pick a version and support those for years, while upstreams usually force you to use the latest version all the time to get security support. With distros _I_ get to decide when I upgrade, and the reduced frequency is a nice bonus. Having a single entity for all software on the system is also valuable, as there's just one tool to learn and one place to check for updates.
- 5y ago

cjfd 5y ago

Perhaps the distributions would be more inclined to include up-to-date version if the standard in the python community was not to break everything all the time. There are distributions that keep up-to-date, though, e.g., archlinux. Serving the needs of python developers vs. sysadmins is a false dichotomy. Python developers develop on a system that they need to admin. One great thing about linux is that everything on a system can be kept up-to-date using just one software tool (a package manager). You are not going to convince me in a million years that this is a bad idea. Now, for things that are needed in development you sometimes need different versions, in particular if you happen to have an OS that ships very old versions. For that purpose there should be some sort of tool and not a gazillion tools that are all incompatible and behave in slightly (or not so slightly) different ways. 'python -m venv' is different from 'virtualenv'? WTF? Also, if your distribution is up-to-date (e.g., archlinux) and you are pinning to older version there is also a problem. As the article puts it 'pin their dependencies to 10 versions and 6 vulnerabilities ago'. And if you actually want to maintain your software in the future you, at some point, have to go to the new version anyway.

BiteCode_dev 5y ago
Sure, so my lib works on mac, windows, centos and debian. Tell me how I'm supposed to define my lib python deps ? How am I suppose to package the lib ? Distribute it ? Deal with os versions differences? Allow isolation for my user projects ?
- 1337shadow 5y ago
  I'm sorry, doesn't pip already works there? Otherwise, there's pyinstaller which is great but it requires an entrypoint so it won't do with a pure lib that doesn't expose any script at all.
  - BiteCode_dev 5y ago
    Yes that's the point. We won't use distro packages because only regular python packages allow this. Edit : arf, sorry, I answered the wrong comment, I actually agree with my parent.

rollcat 5y ago

> Python packaging needs to better serve the needs of python developers, not those of sysadmins. Please substitute s/sysadmins/users/, and realise your developers are users too. I've been doing Python for almost 15 years; and I'm getting really fed up with some things. Packaging is a mess. Distribution is a mess (for servers/IoT - Docker saves the day; for desktop - I feel like giving up). Managing the installations is a mess; upgrades can be impossible - I'm hard-stuck on Python3.6 on one project! I find myself rewriting many smaller tools in Go or Rust, just because I can upgrade the toolchain at any time, and/or ship a static binary. But Rust has a very high barrier to entry, and Go tends to be simplistic. I'd fully jump ship today, but Python has just too much momentum behind it.

ShamelessC 5y ago
I agree to some extent, but I believe they were mostly trying to convey that much of the work is done by volunteers who are largely only motivated/have the resources to test a small subset of potential deployment scenarios. Those contributions are still valuable and contributors themselves don't owe you anything. The best way to fix this issue would be to get involved (or switch languages as you suggested, that's fine too).

drran 5y ago

> Python packaging needs to better serve the needs of python developers, not those of sysadmins. Use Fedora, or Arch, or another developer-oriented distro.

goodpoint 5y ago

> Python libraries shipped by distributions are so old that this mechanism is mostly useless for python development. Absolutely not. Most distributions ship versions that are released before the distro release freeze. The whole point of using a distro is to have a reliable and trusted development and production platform.

regularfry 5y ago

Yes, exactly this. Only I view it from the other way round: to try to do development with distro-supplied language packages is a category error. Distro-supplied interpreters and their associated libraries are there for the applications supplied and supported by the distribution. Unless you are developing something to be part of the distro, they are not for you. Do not let the distro get between you and your libraries. Supply your own.

Pokepokalypse 5y ago
>Unless you are developing something to be part of the distro, they are not for you. . . . forgetting the original purpose and mindset behind Linux in the first place. From a 1990's point of view; it was always intended to be a hobbyist OS; and in most cases, one had to compile all the binaries and kernel one's self. There was no such thing, really, as a "user" or "admin" - everyone was considered, and expected to be a developer.
- regularfry 5y ago
  Yes, and distributions arose to solve the problem that plagued early Linux: how to have a set of applications and libraries with consistent and mutually compatible versions of everything. If you're compiling everything yourself, building it so it all works once is an achievement. Keeping on top of as many moving targets as there are binaries in the system so that everything keeps working over time is not practical once the number of moving parts gets high enough. Distros solve a real problem, but the trade-off is that some parts of the system must exist to serve itself.