29 ms·

Python: Please stop screwing over Linux distros

• arghx 5y ago
  The PSF won't fix it, its members are the problem. Dozens of people profit from useless churn that results in billable hours, and they actively purge anyone who opposes. Switch languages, this is a lost cause.
• Havoc 5y ago
  And then you’ve got the MS store that just hijacks any calls to python completely even if you installed it separately Debian apt installs with using python3 and specifying pip as a module call is the only thing keeping me sane
• drcongo 5y ago
  Anything trying to use that risible xkcd panel to make a point is instantly, easily dismissed.
• dekhn 5y ago
  I just spent a bunch of time with pip yesterday that was best solved with apt install python3-psycopg2.
• deleted 5y ago
  [deleted]
• elzbardico 5y ago
  > I manage my python packages the only way a think it is sane: by using my Linux distro package manager. Your honor, I have no more questions for the alleged victim. <drops mic>
• noodlesUK 5y ago
  I am not one to install python packages using my distro's package manager, but I totally agree with the sentiment that we need a more standard build/dependency management system in python. I like poetry, and I think most people are heading that way, but it doesn't seem to play super nice with pyenv (which is a critical tool) a lot of the time, and I think that a first party endorsement of the "one true build system" a la golang or rust would be a huge step.
  • hkt 5y ago
    Golang is a great model to follow. I think the fundamental thing that python (and ruby, when I still used it) has taught me is to run a mile if a language is without a robust dependency management system. The pain is just not worth it, even for a nice language with an otherwise robust ecosystem.
    • yxhuvud 5y ago
      Huh? The package dependency system in Ruby is a breeze, and has been a solved problem since 2007 or whenever Bundler was made.
    • noodlesUK 5y ago
      With golang, it's not just the dependency management side, gofmt (and various bits of other tooling) is also an incredible blessing. Yes, prescriptivism can sometimes feel weird to us computer nerds, but sometimes it's just much better to have a canonical way of doing common tasks. It means onboarding people is much easier, and infrastructure/code/whatever is more reusable.
    • goodpoint 5y ago
      Go is terrible for distributions to package due to the poor library versioning. Also, static linking makes security updates really expensive to build at scale.
  • withinboredom 5y ago
    I’m not a Python dev. I do need to occasionally run things written in Python. I made the mistake of trying to get pip + Conda + pyenv (or whatever) to install a fairly simple tool. I have no idea how the dev got their setup working, but it was totally and utterly unreproducable, even after they sat down on my computer for several hours. In that amount of time, I could have probably rewritten it in PHP (that’s actually exactly what I ended up doing while they attempted to get a Docker container running). Needless to say, I will only use the distro package manager these days. I know the versions are (probably) compatible, maintainers will usually backport security patches, etc. You get none of that using whatever flavor of the week python package manager.
• pipeline_peak 5y ago
  And stop putting everything on the planet into a god damn virtual environment.
  • rbanffy 5y ago
    Virtual environments are a good way to have different packages for different applications running on different versions of Python. My usual rule is you don't mess with the system-provided Python environments for specific applications you are working with. I would even suggest dropping support for many packages at the distro level unless they are required by other non-python packages (the same way Django and Twisted are requirements for MaaS).
    • 1337shadow 5y ago
      > don't mess with the system-provided Python environments for specific applications you are working with Right, but then just use pip install --user instead of a virtualenv. Actually --user is the default now when running pip install from non-root user, so, just pip install as a user will work and not mess with the system packages, just don't do sudo pip install.
      • gjulianm 5y ago
        I've run into problems when running pip install --user. Someone installs a package in the user environment, and that works. But later you'll install a package in a shared virtualenv (because a Python program needs to run as a service or by another users or whatever) and pip doesn't install it because it's already on the user environment. However, other users don't have access to that environment and they will have an import error that you won't see from your user. Not to mention that installing every dependency in the same environment is a recipe for both disaster with version conflicts and bloat when you don't really know which packages belong to which applications.
        • 1337shadow 5y ago
          Interesting problem but I think it's more a problem in virtualenv, which don't use global site package by default, I'm surprised that it uses user site packages by default. However, installing every dependency in the same environment is also what distro package managers do, I don't see that as a recipe for disaster, it all depends how well the said packages are maintained in which case just upgrading them should fix whatever problem.
• jjgreen 5y ago
  I've had success restricting myself to Python packages provided by the OS (Debian) in production. The only down-side is there is only one Python (well, 3.* or 2.7) against which to test, so I drop back to a pip install of the same packages for CI, and that's where 90% of my Python pain comes from, every maintainer of every Python infrastructure package feels entitled to shout "DEPRECATION, do THIS, do THAT, use THIS" at me. It's ill-mannered and unpleasant.
• geofft 5y ago
  This is a very weird post to read, as someone who does (occasional) distro Python work - I feel like Python absolutely is listening to us! It's just that it's hard work, and distro packaging is mostly done by volunteers, and there are a whole lot of things to work through. Here's some work I and others did earlier this year, which I thought was a great example of folks from the core Python packaging world and folks from distros working together: https://www.python.org/dev/peps/pep-0668/ https://www.python.org/dev/peps/pep-0668/ See the massive table of use cases for all the things we had to think about. You might note that one of the things it needs is additional participation on the Discourse thread from the authors (like myself) and from other distros. Again. It's mostly work done by volunteers, and there's a lot to work through. There's no magic to it. I can tell you that the following (from TFA) will absolutely make things worse for distros, though: > I call on the PSF to sit down for some serious, sober engineering work to fix this problem. Draw up a list of the use-cases you need to support, pick the most promising initiative, and put in the hours to make it work properly, today and tomorrow. Design something you can stick with and make stable for the next 30 years. If you have to break some hearts, fine. [...] These PEPs are designed to tolerate the proliferation of build systems, which is exactly what needs to stop. Python ought to stop trying to avoid hurting anyone’s feelings and pick one. If you want the PSF to fund some engineering work on its own that it can finish much faster than any volunteer packager can even read the proposal, break some hearts, stop proliferating build systems, and hurt people's feelings, they will absolutely do that and say "Distro packaging is not supported, we only support virtualenvs. Users should make virtualenvs. Distro software should ship virtualenvs. Installing a Python package systemwide is meaningless." That's clearly the best-supported option right now, and it's a surprisingly technically defensible answer, but it's not going to make you happy.
  • turminal 5y ago
    "use virtualenvs and install your dependencies from an unfiltered, unsupervised, untrusted source" is certainly the solution most aligned with the rest of the industry but I struggle to see any technical benefit from it. Other than perhaps "move faster and break more things".
    • 5y ago
• mangecoeur 5y ago
  Everyone complained, no one addressed the fact that for years the whole of python packaging was handled by like 2.5 people. But as a rant this, like most of the ‘but just fix it’ rants, fails to acknowledge the hugely diverging needs of different users. I could not live without conda, since it’s the only sane way to get a working recent geospatial stack. Others need to run embedded environments, or portable ones, some need long term stability while others need bleeding edge packages that haven’t been released yet. Solving for a single case is straightforward; solving for all of them , not so much.
  • microtonal 5y ago
    And yet, other languages ship package managers [1] that are widely loved within their ecosystems and cover everything from microcontrollers to server applications. I think what it makes it more difficult in the case of Python is that it has decades of legacy to deal with. No consistent semantic versioning, packages that expect that they can modify their package path in-place (this is a nightmare for immutable systems, like NixOS or OSTree-based system like Silverblue), a wide variety of build systems that sometimes hook into make, etc. Solving this is a hard problem. This is why an authority like PSF has to step in and say: this is how it is going to be done from here onwards. [1] E.g. Rust's Cargo.
    • komuher 5y ago
      Kinda diffrent Rust its like 3 years old when it start getting popular. Python is 13+ years old when it starts getting very popular. You cannot compare legacy to modern solution -.-
      • smallerfish 5y ago
        Maven was launched 9 years after Java got popular. Everybody was using Ant, everybody decided Ant sucked and moved to the new solution. While Gradle is the new kid on the block, it keeps the infrastructure of dependency management and is thus compatible with Maven to that extent. It can be done.
• nerdponx 5y ago
  > Every one of these package managers is designed for a reckless world in which programmers chuck packages wholesale into ~/.pip, set up virtualenvs and pin their dependencies to 10 versions and 6 vulnerabilities ago, and ship their computers directly into production in Docker containers which aim to do the minimum amount necessary to make their user’s private data as insecure as possible. Is this any different from any other programming language ecosystem? Is Python really doing worse than Node, Ruby, Perl, Lua, Go, or Haskell in this regard? Python files go in `/usr/lib/pythonx.y`, Python finds said files, programs run. Yes, Python build and packaging in general is messy. But I am curious why, specifically from a distro perspective, it's any worse than anything else.
  • pm215 5y ago
    The notable language ecosystem which is different is "C on Unix", which grew up with "the system provides the dependencies, it's often easier to avoid a dependency if you can, the program usually needs to be able to cope with whatever version of the dependency the system has, rather than pinning to a specific one". That's the primary ecosystem that most Linux distro package managers developed with as their platonic-ideal-shape-of-an-application, I think.
  • BiteCode_dev 5y ago
    There are some differences indeed: - python is more active than most alternatives, you have new packages created every day. - python is massively used outside of the web, unlike JS, ruby or PHP, that are 99% web. You get Python in SIG softwares, data analytics, automation, pen testing, sysadmin, biology, etc. It's a huge graph. - python is used by the distro themselves to code features of the OS. E.G: you remove Python, there is no yum. - python has a rich compiled extensions ecosystem, produced from c, c++, fortran, and assembly. It's very complicated to ship them. - it's much more common to have several Python installed than for other dynamic languages. So isolation matters even more. So the difference is the sheer size of the problem.
    • nerdponx 5y ago
      That's a good analysis, thanks. But I don't think any of it justifies the ire towards Python, its community, or its devs that was expressed in the blog post. I also do want to point out that there are quite a lot of general-purpose CLI tools written in Perl, Ruby, and Node.
• tuldia 5y ago
  Am I the only one that is not having issues with python and distributions in general? I get all my dependencies from Debian and they all work, when I need something that is not yet packaged, I use pip. What are people doing to get all this issues? I don't understand...
  • dolmen 5y ago
    Looks like the "Works on my machine" syndrome. What about working on a project with a team? What about deploying the code on another machine?
    • PeterisP 5y ago
      When deploying the developed application on some server, all the exact dependencies get installed there. The main reason for the existence of the server and its configuration is to run the application, so the server adapts to the needs of the application and gets the dependency versions preferred by the app, instead of the application trying to adapt to the server and trying to make do with the libraries already existing there.
    • 1337shadow 5y ago
      Then I go with something like docker-compose
      • __float 5y ago
        Docker is a very heavy-handed approach to mitigate more fundamental issues with Python's package management, don't you think?
        • 1337shadow 5y ago
          I'd say it's a heavy-handed approach to mitigate more fundamental issues with how python packages are maintained, if everybody wants to pin different versions then we're going to have to install different versions of everything which is what npm does and I consider that heavier. Again, it's all a question of point of view, what we see as a package manager problem and causes us to keep reinventing packages managers, might actually be a problem with how we maintain our packages, my point of view being the latter. But I'm digressing. When it comes to installing on "another machine", you don't know what Python they have, you don't know what libc they have, and so on, that is exactly what containers attempt to mitigate, so that seems exactly like the tool to use for this problem.
• ivirshup 5y ago
  Can someone make the case for distributing python packages at the distro level to me? Especially scientific software for data analysis? Our project gets a few issues opened by distro maintainers who have trouble with some part of their build process. Is it really worth our project maintainers time to help troubleshoot esoteric build processes when we already provide source, wheel, and conda distributions?
  • nerdponx 5y ago
    I don't know what your project is specifically, but it's reasonable that a system-level user-facing application might want to depend on Numpy or Scipy.
  • Sesse__ 5y ago
    Users generally don't want Python packages; they want software. They don't care what it's written in, and shouldn't have to install it differently depending on what you chose. “Just learn $language_package_manager_of_the_day, and hope it doesn't break anything when your system changes” is the equivalent of the 90s “./configure ; make ; make install”, and we should have moved past that for end-users by now. Most users are not developers, and installation procedures need to cater to both.
    • ivirshup 5y ago
      But our package is for scientific data analysis. It’s a python library. If you’re distributing an application, shouldn’t you just ship an environment with our package bundled?
      • rbanffy 5y ago
        > shouldn’t you just ship an environment with our package bundled? That's how I would prefer it, but, if the intended form of distribution is a distro package, then it should pin itself to the versions the distro provides and avoid (or vendor in) packages that aren't available. It is possible to just place everything in the app directory and distribute it this way, but it's kind of ugly (and doesn't pick up security updates from the distro)
• robertwt7 5y ago
  we need one, only one build system, not many. Feels like atm python is a bit shattered around like c++ Using JS or rust seems easier with the unified build system.
  • uranusjr 5y ago
    Is your unified JavaScript build system using NPM or Yarn? ;)
    • robertwt7 5y ago
      both works either way with 1 package.json. doesn't that feel good?
      • uranusjr 5y ago
        I am happy that it actually works identically and interchangeably across tools for you.
• 1337shadow 5y ago
  > pin their dependencies to 10 versions and 6 vulnerabilities ago That is the real problem IMHO. Most python users like to pin dependencies so that "their program don't break", and that's also the reason why so much effort is put in what they call "correct" dependency resolution resulting in the creation of new python package managers that all do "more correct" dependency resolution and make programs "break less" and at the same time require less efforts from the maintainers to actually maintain their code by doing dependency upgrades. And then one day you want to upgrade a dependency and you realize you're 10 releases behind on 10 dependencies, and what was supposed to be a quick maintenance task is now 100 maintenance task. If you don't pin, your program will break one day or another, a user might open an issue with the traceback, or, your program will break directly in CI where you will see the traceback. Upgrade it, or contribute to the dependency, but just go ahead and fix it, instead of being defensive and trying to have dependency resolution that "doesn't break". At the same time you'll be adopting the actual practice of "Continuous Integration", of your dependencies, which has a better cost/benefit ratio. I always avoid pinning dependencies, I try to make pip just install the latest version of everything, I am willing to contribute upgrade fixes to any Python package I use, but some dependencies do pin which breaks my own aggressive Continuous Integration practice, heck, I'd even need an option for pip to ignore version resolution at all so that I can make all my contributions to upgrade everything I use. I even remember when I had CI test matrix with all combinations of versions of everything, I don't do that anymore, I just support the latest of everything, we can always have the latest python with containers anyway so that's not even a blocker anymore. If you're not a "techbro using containers", it'd be fine too because you should then be able to make your distro packages at any point in time and expect all of them to work together, minus the delta of the handful of upgrades that are pending to release here and there.
  • pimterry 5y ago
    > I always avoid pinning dependencies This is an very quick route to a maintenance nightmare imo. If you have totally unpinned dependencies, and you come back to a project after a year untouched, or 5 years, and it no longer works - which dependency update broke it? I don't agree that using an outdated package is necessarily a problem at all. Some versions are done! You don't need the latest version of every possible package. You don't necessarily need to update _ever_ (which is why this differs from CI). These updates are often entirely unnecessary churn. There absolutely are vulnerabilities in some old versions, and those updates are necessary (but tooling & notifications to easily handle this have dramatically improved in recent years, especially on GitHub). There will also be vulnerabilities in new packages though, which may be unknown, and will often not exist in older much simpler versions. Using a well-tested version of a dependency that does exactly what you need is not less secure than chasing the latest version at all times without a specific reason. I've found manually updating packages on the rare occasions where relevant vulnerabilities arise, and using existing working versions without changes the rest of the time has been perfectly effective over many years now, and avoid the shifting sands of external dependencies wherever possible means that a project that worked 5 years ago still works _exactly_ the same today. I rather see more software go in this direction, valuing reproducibility & known correctness (i.e. with isolated pinned dependencies, in some form) over 'always be latest' dependency updates and the complex & hard to reproduce bugs that those shifting dependency interactions can create.
• StreamBright 5y ago
  Interesting take. I use Python daily and virtualenv is really good for our use cases (running CI/CD, testing installation and running in production in a container). I am not experiencing too many problems with pip + venv. We also maintain our own libraries and even those were relatively easy to set up. Mypy and yapf also helps to maintain a style and type correctness (do not pass in None accidentally).
  • rickspencer3 5y ago
    I do all of my server side Python development in a container. Whenever I start a new project, I create a dockerfile as a first step. This is the easiest way I have found to make my code portable. The main downside I have found is with IDE support. I use the latest tag, so I get updated containers periodically, which has not been a source of breakages. I have not figured out how to tell VSCode to use the Python in my container for statement completion and validation. For personal projects on my desktop, I have given up and just use pip install, and then don't try to share those scripts.
• fer 5y ago
  This is my decision flow and I rarely have an issue: Are you the end user of the Python code? Yes -> Is available in your distro? Yes->Use package manager No->Use pip install in user mode No -> Create virtualenv with the Python version you want (including pypy!) and do your pip thing there Some extreme use cases may benefit from anaconda, but personally I've never needed to use it. My only pain point is dealing with legacy code that relies on PYTHONPATH. Nothing good ever starts setting PYTHONPATH.
  • nerdponx 5y ago
    > Create virtualenv with the Python version you want (including pypy!) and do your pip thing there You might benefit from using Pipx in this case: https://pypa.github.io/pipx/ https://pypa.github.io/pipx/ Pipx is good for the case of "I want to run a standalone Python application that is available through Pip, but not my system's package repo." This is a more common case than you might think. It's a sensible alternative to `pip install --user`, and having self-contained deps for tool is a bit like `npm install --global` or even `volta install`.
    • WhyNotHugo 5y ago
      Yup, this kinda works. It doesn't address the greater issue tho: that it's getting harder and harder for distributions to package things right, and provider packages for their users (evidenced by the fact that you need a second package manager just for python stuff).
      • nerdponx 5y ago
        How is Pip any different than Cpanm, NPM, Gem, Luarocks, Nimble, Go's thing, Cargo, whatever JVM people use, whatever Haskell people use, etc. in that regard? Distros have a hard job, but at the same time programming language tooling devs have more "customers" than just distro maintainers.
  • isolli 5y ago
    Hm, why? I'm a happy user of PYTHONPATH!
• jimnotgym 5y ago
  It seems a bit weird to criticise lots of people for trying to solve the dependency problem. I know lots of people hold up npm as the standard. Npm came out around 10 years ago, Node around 12. Both of which had the benefit of hindsight at how the problem had developed for other languages. Python came out 30 years ago and pip came out 20 years later. Of course there is a lot of stuff left behind. What do people suggest, a breaking change, maybe? Python 4?
  • m4rtink 5y ago
    I'm not sure NPM is a good example of anything, with its insane dependency chains and the barrage of security issues.
  • georgyo 5y ago
    Npm is absolute hell. One thing that people do like about JavaScript is that you can import multiple versions of the same package. This solves that depA requires 1.0 and depB requires 2.0. However this makes auditing nearly impossible. Small projects can end up with well over a thousand dependencies that is just unmanageable.
  • selfhoster11 5y ago
    NPM is the perfect example of overly-atomised dependencies. No project should have a dependency graph numbering in the hundreds or thousands of nodes. IMO, the Java or APT packaging ecosystems solve the issue in a far superior way.
• elurg 5y ago
  Python libraries shipped by distributions are so old that this mechanism is mostly useless for python development. This also applies to many others programming languages which have their own packaging systems. While python packaging is indeed messy the needs of traditional linux distributions are by far the least important. Python packaging needs to better serve the needs of python developers, not those of sysadmins. Having a "linux distro" interpose itself between you and your libraries is a fundamentally broken system not worth fixing. All decisions related to dependency choices fundamentally belongs with upstream.
  • dgan 5y ago
    Exactly agreed on that. I don't understand what value distribution is providing by repackaging python libs, they re always way too old to be usable, and they re global while I work on many projects, with their own incompatible requirements. Maybe I am dumb, but I exclusively use virtualenv and pip..
    • avidphantasm 5y ago
      I agree with this for pure Python libraries. However, as soon as you get into things that bind to C libraries (Numpy, GDAL, etc.) it quickly becomes much easier to use the package from your distro.
    • rlpb 5y ago
      > I don't understand what value distribution is providing by repackaging python libs... I want to easily and safely use some app my distribution ships. I want to receive security updates automatically for all such apps. I don't care what language it's written in or what its dependencies are. These app packages provided by the distribution have dependencies that are also packaged by the distribution so that dependency resolution works. Since the point of a distribution is that it can run apps, the value is that a distribution works at all.
      • dgan 5y ago
        Valid remark. I had "development side" point of view, not "usage"
• CornCobs 5y ago
  I once attempted to install a program that was distributed via pypi (pip was the only way I could install software on that server). The shebang in the script file was #!/bin/python. Thats python2 on most Linux distros and so it couldn't run
  • 1337shadow 5y ago
    That's weird, the console_scripts hook for python packages generates a script with the same python in the shebang that pip is executed with to install the said package.
• rbanffy 5y ago
  My usual approach is to let the distro install the packages it wants and then, for each thing I'm working on (I sometimes write Python code for a living) can have a separate environment with different module versions. This makes sense when my artifacts are deployed via Docker images that are generated with `pip install`, but not as much if you plan to install them in your machine. The only thing I build like this for public consumption is pip-chill, which is not intended to be used with the bare Python environment of the distro (I probably should add something that makes it refuse to install that way). Even when the things are not Python-based, I often use a virtual environment, managed with virtualenvwrapper (installed on the distro level). One example is a Terraform config that relies on some Python tools to manage deployments - the Python tools are local to that virtual environment and not usable anywhere else. If I need to develop something that'll need to run with the distro directly (something that could be distributed as ditro packages) I'd use a virtual environment and tailor the package versions in requirements to the ones available in the distro. This way, multiple distros can be addressed with multiple environments pointing to the same source directory and multiple requirements files for tests.
• dikei 5y ago
  Currently my workflow when starting a new Python project is: * Create a new conda environment with the minimum python version I need to support. * Use poetry to install dependency * Package the production build into container image for deployment. The only problem I have is sometimes Poetry choke on compiling libraries with C-dependency, luckily most libraries provide binary wheel now.
• cdent 5y ago
  I'm always confused by these sorts of posts because they happen often so there is clearly a problem but for some reason I've never had much of an issue. I've been using and developing with Python for about 15 years. In that time I've worked on Python projects large (OpenStack) and small (gabbi) and taken over maintenance for some old standbys (wsgi-intercept, paste to name two). Dealt with the 2->3 transition. Released a whole bunch of things to PyPI and relied on far more things that I've pip installed from there. It's been fine. I don't know what I'm doing differently.
  • sufehmi 5y ago
    Just scroll down - loads of horror stories shared by others
  • 1337shadow 5y ago
    OpenStack uses pbr which they created and which is pretty cool. I use setupmeta for the same purpose but could have used pbr. https://github.com/openstack/pbr https://github.com/openstack/pbr https://github.com/codrsquad/setupmeta https://github.com/codrsquad/setupmeta
  • WhyNotHugo 5y ago
    So there's a few types of projects you can write in Python: 1. Server applications that run in a dedicated environment. 2. Tools you write and run just on your machine (or some virtualenv, whatever). 3. Redistributable cli or desktop applications which end users will install and use. For the first two types, you should never have any issues with Python and its dependency situation. You pin everything, and that's it. For the third kind tho, it's complete pain. Different distros ship different Python versions, so you need to support all of them. You also have to consider that dependencies can't be an EXACT version, you have to support a range of them, and a variety of combinations. And then, one dependency has a version that works in Python3.6, and another for 3.9. But they had an API change, so which one do you use? It'll break for half your users either way. Of maybe just put some `if version <= 3.6` all over the place, like we did during the py2->py3 transition?
    • deleted 5y ago
• smallerfish 5y ago
  Yes, python packaging is a mess. And agreed, there are two separate use cases: development and using the software. But, are distros creating too much work for themselves by trying to package every itty-bitty python library (and for that matter, every npm library)? Are distros doing anything more than scanning CVE databases with the library versions, or are they _actually_ auditing the versions they choose? (Not that there's much choice, since python also has a shitty story when it comes to backwards compatibility; if you're going with 3.10, there's possibly only one version of a given library that will work.) Java has a commonly used "fat jar" approach which rolls up all dependencies into a single file. It's excellent. In the python world, this doesn't exist, because virtualenvs aren't portable. If that can be fixed (perhaps a specific section in requirements.txt that captures anything that needs to compile C for the platform) then a distributable virtualenv would become possible. Distros would then scan the application for vulnerabilities (via requirement.txt's manifest), build the distributable-virtualenv, and ship _that_. Python library maintainers don't have to do anything different (except, of course, use the standard way to declare dependencies).
  • keymone 5y ago
    there's https://github.com/spotify/dh-virtualenv https://github.com/spotify/dh-virtualenv
    • smallerfish 5y ago
      That's cool. Anybody using this? How robust is it?
      • keymone 5y ago
        it's decent. though these days building container and shipping it with a `docker exec` shim is much easier.
  • deleted 5y ago
    [deleted]
  • nerdponx 5y ago
• globular-toast 5y ago
  I think there is an increasing trend for developers to get confused between their tools and their output. These are your tools: * pip * pip-tools * pipenv * poetry * virtualenv * venv * setuptools * pyenv * requirements.txt * etc. These are your outputs: * wheel (binary), * tarball (source). The tools in use are completely irrelevant and only add noise to the discussions around packaging. So what actually is the problem here? Are projects delivering broken outputs (ie. bad packages)? Are they not delivering outputs at all? Those are real problems but pointing at the number of tools that exist is not helping.
  • pxc 5y ago
    Python itself confuses sources and outputs in that not every package has consumable source tarballs— for some packages the only way to get static metadata on dependencies is by starting a build as if to get a binary output. Native dependencies are not really handled except in an ad-hoc way, either. If you want to import Python packages into a real packaging system, you are confronted with the tools whether you want to be or not.
• captainmuon 5y ago
  Distros, please stop screwing over Python packaging. It is incredible that Debian/Ubuntu pick Python apart and put modules into different packages. They even create a fake venv command that tells you to please install python-venv. What they should just do is offer a bunch of packages like python3.7, python3.8 that install the official python package wholesale into /usr/python or someplace and then symlink one of them to `python`. If I would get to redesign package management (both for Linux distros and for languages), I would have one package manager that installs everything into a global package cache, and then pick the correct version of libraries at run time (for Python: at import time). Get rid of the requirement that there is only one stable (minor) version of a package in the distribution at one time. This has become unworkable. Instead, make it easy to get bleeding edge versions into the repositories. They can be installed side by side and only picked up by the things that actually use them.
  • hk1337 5y ago
    That sounds similar to what I do in macOS. I hate installing homebrew to /usr/local so I started installing it to ~/.brew and I hate using the python from homebrew so I always use pyenv.
  • rizkeyz 5y ago
    This is the real legacy problem: Python comes from a world, where only one version of one packaged seemed the right way to do. I do not have a good idea, but other ecosystems evolved much more sane in the realm of packaging. While not ideal, Go has done a fairly good job - and the "module" operations are instant - which they should be.
    • 1337shadow 5y ago
      Ok but Go compiles statically, while you can do the same with pyinstaller, I don't think that's really comparable as we're talking about deployment right there.
      • rizkeyz 5y ago
        Static binaries are a different story. Go has dependencies as any other modern language and they had a bad story in the past and have a better story today. Sketch for python: Create a ~/.cache/python/packages directory. Manage all dependencies there. Make the python interpreter "package aware" so that required dependencies are read off a file from the current project (e.g. "py.mod") and adjust "system path" accordingly and transparently. Or something along those lines. No extra tool, a single location, an easy to explain workflow (add a py.mod file, add deps there with versions, etc). I'm just thinking out loud, but it does not need to be hard.
• robertlagrant 5y ago
  A suggestion: 1) If your distro requires Python, don't put it on the path. Refer to it another way if you need it, e.g. have a distroname-system-python package you upgrade at your own cadence. 2) That's it. Then developers can install Python how they like, and it's (probably) all fine.
  • cozzyd 5y ago
    Indeed, many are moving in this direction (https://developers.redhat.com/blog/2019/05/07/what-no-python-in-red-hat-enterprise-linux-8 https://developers.redhat.com/blog/2019/05/07/what-no-python...)
• foxes 5y ago
  Nixos/Nix solves all python packaging problems.
  • trulyrandom 5y ago
    Nix solves a lot of Python packaging problems and I'm a happy user. But you lose some of the convenience of plain pip. For example, with Nix, it's no longer easy to mix and match versions of different packages. You just get the package versions that happen to live in the snapshot of nixpkgs you're using. If I want to use an older version of a package than the one in nixpkgs, I now have to add an override for that package and pray that changing the version/hash is enough.
• riccardomc 5y ago
  The success of Python and Go shows that people don't care as much about what distributions maintainers feel is "quality packages". The scale and quantity of software being developed is just too much for traditional Linux distributions approaches. I personally rely on Debian to know that most of the packages I use are reasonably secure. But often I need to sacrifice flexibility and bleeding-edgedness. And it couldn't be otherwise. There are 339,267 projects on PyPi.org. I bet a good percentage of these are riddled with security flaws. But people use them anyway. How many distro maintainers would you need to handle this workload? Is it worth it? Does anyone care? It looks like people care much more about experimentation and speed of development than stability, security or coherence and cleanliness of the solution. The author seems to feel this is wrong from an engineering (or even moral?) perspective. I am also uncomfortable when I have to deal with Python and I have my own favourite few tools... but perhaps if Python users really wanted a single solution then it would already exist. If you are instead convinced that there is this need and no adequate solution, then congratulations, you just found a gap in the market. Go on and do better than everyone else before you. Relevant XKCD comic is already in the article...
  • selfhoster11 5y ago
    As an end user, I definitely care that my software works, doesn't crash, and doesn't leave my machine wide open for attacks. I don't want things to be this chaotic
• BiteCode_dev 5y ago
  The day packaging for distros will be easy, we will use it. Right now, making a deb is hard. Isolating 2 projects with different deb verions is hard. Distributing debs is hard. Upgrading your OS but not your python debs is hards. Then rinse and repeat for red hat, arch, nix, mac and windows ? Yeah right.
  • eigengrau 5y ago
    Since Nix packages are distribution-independent, once you have it packaged with Nix, you could theoretically skip packaging for Ubuntu etc., but of course that may raise the bar of entry for your users.
    • BiteCode_dev 5y ago
      I doesn't work on windows, so you lose half of your users. And WSL is not the answer to that. Also, unless you repackage thousands of compiled c extensions and you play well with anaconda and can plug into the entire python ecosystem of platforms such as heroku, python anywhere, databrick and so on, you then lose 90%
• lmm 5y ago
  > What is it about Linux distros that makes our use-case unimportant? Have we offered no value to Python over the past 30 years? Indeed you haven't. Worse, you've actively damaged Python's efforts to improve. I mostly work on the JVM these days, and I think one of the main reasons dependency management there is so gloriously simple and effective is that the Debian packagers weren't around to fuck it up.
  • lvass 5y ago
    There is plenty of Java packages in APT.
    • lmm 5y ago
      There are now, but for a long time they had to live in contrib because Java wasn't open-source, and so apt didn't deeply integrate and customize their package management the way it did with perl or python.
• toyg 5y ago
  The flexibility the author decries is one of the strengths of the ecosystem. Oh, things don't work for datascientists on Windows? Here, use conda. Bunch of different webapps on same server? Use venv. And so on and so forth, every niche will use what works best for them.
• kevinmgranger 5y ago
  Distro maintainers say that language ecosystem packaging makes it hard for them. Language ecosystem packaging maintainers say that distro package managers make it hard for them. The year is 2021 and there is no work towards synthesis. It will be endless, fruitless yelling from each side. Maybe 2022 will bring change. I'm not holding my breath. To anyone who finds themselves on a single "side" in this argument: if you have ever said "why do you need to do that" as an accusation instead of with curiosity, you exemplify the problem.
  • katbyte 5y ago
    Question is are the other languages saying the same thing about the distros? Or is this just a python thing
    • Macha 5y ago
      Rust/Go/Node mostly just ignore distro packagers entirely. Java sometimes has these issues, but mostly distros aren't going around splitting up fatjars for desktop applications or wars for Web applications. C# just gets shoved into opt as it was considered a problem child to be left alone for a long time. Perl has avoided this problem by stagnating at the cost of its long term userbase. C/C++ mostly get along fine since the distro package model was built for them, but occasionally someone gets upset about meson or ninja or something.
• ReleaseCandidat 5y ago
  There actually exists a solution to unite the distros packages with the developer ones: Nix (or GNU Guix, if you prefer that). And yes, both 'are' a Linux distribution too. https://nixos.org/ https://nixos.org/ https://guix.gnu.org/ https://guix.gnu.org/
• scotty79 5y ago
  > Every one of these package managers is designed for a reckless world in which programmers chuck packages wholesale into ~/.pip, set up virtualenvs and pin their dependencies to 10 versions and 6 vulnerabilities ago, and ship their computers directly into production in Docker containers which aim to do the minimum amount necessary to make their user’s private data as insecure as possible. That's pretty sobering look at the state of affairs.
• thinker5555 5y ago
  As a "self taught and still learning developer-lite", I love the python language, but the ecosystem drives me nuts. I feel a lot of the pain expressed in the article, and it pretty much speaks to my current conclusion of "I'm trying to do things the 'right way' but there doesn't seem to be a 'right way'". I've seen a few comments here about how Nix/NixOS fixes the whole python binary/library mess, but I'm having trouble understanding how. Does anyone have any insight to share about that? Additionally, the whole thing kind of makes me want to move away from python wholesale. I was wondering if there are other languages that are great general languages like python that don't suffer from this whole packaging and versioning mess. Ruby? Go? Something else? I'm looking for something high level, somewhat easy to learn, and with good library support for things like working with databases and tabular data. Though I don't know much about them, I just feel like I don't want something like Java or C++ or anything like that. I want to "get things done" and not have to worry about tons of boilerplate or working at really low nitty gritty levels.
  • hocuspocus 5y ago
    Scala has its own issues, but it's worth a try. It's easy to learn coming from Python, even more so if you can start with Scala 3 right away. You cannot really get more high-level than that. There are good libraries to work with databases, Quill comes to mind if you want something user-friendly. And you can always fall back to the many Java libraries in the big data ecosystem (even though you should probably avoid it if you can).
  • toyg 5y ago
    > I've seen a few comments here about how Nix/NixOS fixes the whole python binary/library mess Nix is a bit of a cult. Its theoretical aims are laudable, but in order to get there it forces you to do a lot of work and reason strictly in its own way. Whether all this work is worth the rewards, I think is open for debate.
    • lvass 5y ago
      Chemistry is a bit of a cult. Its theoretical aims are laudable, but in order to get there it forces you to do a lot of work and reason strictly in its own way. Whether all this work is worth the rewards, I think is open for debate.
• rcarmo 5y ago
  I have a lot of (possibly controversial) issues about this post, for various reasons: - Debian/Ubuntu Python packaging is actually pretty comprehensive, and Ubuntu LTS ships with reasonably updated (although admittedly not latest revision) third-party packages that are usable out of the box, and that I could just list in Ansible to have usable environments spun up. - Python packaging is hardly a mess. I have been using pip for ages without any issues other than forcing wheel downloads for unusual distros (like Alpine, where musl makes it chancy to use some low-level stuff). But if you're on a mainstream distro with modern pip, wheels just work. - If you're not on Linux (or not on the mainstream), pyenv also just works. I have been using it across several years of macOS releases without any significant issues other than knowing to pass it the required build flags to build out the Cocoa bindings here and then (which is easy to do with the brew pyenv). And, finally, I'm constantly shocked at the number of people who just don't get virtualenvs, or who don't know how to switch python interpreters by using environment variables. I've never looked back, and they were instrumental in bridging the gap between 2.7 and 3.5 while I converted some code across (I never really found that the switch to Python 3 was as dramatic as many people made it, perhaps because the code I handled worked after a single pass with 2to3 and minor tweaks).
  • AtlasBarfed 5y ago
    This is my experience with using any python thing on OSX: - brew install: fails probably - pip or pip3 or whatever: fails probably, if it succeeds, breaks something else - look for program-specific install programs/instructions (like the aws cli for example): maybe works, probably bombs Python's reputation (and sales pitch) among developers is a language for people that don't want to program or learn software development. It's basically the new BASIC. Their packaging and installers only reinforce this reputation. Not that it is sweet roses in javaland or many other language ecosystems. Dependency graphs are complicated, because they are graphs when people want them to be simple trees. As for virtualenvs, why does a user of software that happens to be python need to know that?
    • rcarmo 5y ago
      I have the opposite experience. But you are conflating brew with pip, and they are largely independent even if you use pyenv from brew (which you should). Again, I don’t see a lot of logic or understanding of the toolchain in that comment.
• omnicognate 5y ago
  So here's a suggestion to shoot down. The biggest insight in software dependency management is that applications and libraries are different. Both have dependencies but they sit at different places in their dependency graphs. A library can be used together with other libraries and so cannot pin its own dependency versions (if all libraries did so there would conflicts everywhere) but instead can only specify constraints on its dependency versions (eg >=1.0.0). An application sits at the front of the dependency graph. Nothing depends on it. It can therefore lord it over its dependencies, pinning everything in the graph to specific versions. This only works, though, if it doesn't have to share an environment with other applications and unrelated libraries. Systems like poetry (akin to npm or cargo) allow a "lock file" to be generated with pinned versions for all dependencies, satisfying all the version constraints. Applications must commit this to revision control, libraries can if they want (should IMO). This is great as it allows CI and other devs to use consistent versions. The missing piece is that for an application, the lock file should also be used for deployment of the app. If you can ensure that an application is installed in an isolated environment with the dependency versions from the lock file (i.e. that the app was tested against) then a lot of the pain disappears. So, the suggestion: * Add support to standard python distribution (wheels, pypi, etc) for application packages to specify (in addition to ordinary version constraints) a pinned set of "preferred" dependency versions. * Have tools like poetry/pipenv set these to the lock file versions. * Allow a notion of "application packages", which are required to have this information in. This should work very nicely with tools like pipx that deal specifically with python applications. The relevance to linux distros is that linux distros also should only be packaging applications (and their dependencies). Developers using libraries should be managing them with tools like poetry/pipenv, not the system package manager. If the system package manager could install python applications in their own isolated environments along with their pinned dependency versions, most of the pain goes away for distro maintainers. If an application isn't working with the dependency versions it has specifically asked for, it's a clearcut problem with the application as published, and needs to be fixed upstream. I realise this would be a significant change for package managers, but I think the same model makes sense for other languages with similar tooling and at least some of the work should only need to be done once. Go on then, tell me why I'm wrong :-D
  • 5y ago
• _wldu 5y ago
  This is one reason Go has become so popular. If Python weren't such as mess, people could look past its poor performance, but combine that with the installation and dependency issues and it's just too much.
• TomSwirly 5y ago
  > I manage my Python packages in the only way which I think is sane: installing them from my Linux distribution’s package manager. Actually, this is not sane. If you have any Python dependencies, you should always develop and deploy your code using virtualenv, never by installing packages into the system Python!
• Lapsa 5y ago
  build failed path symlinks something something 2.6 2.7 3.0 github issues 3hours (days) later
• coldtea 5y ago
  >The Python community is obsessed with reinventing the wheel Literally, as one of the reinventions is actually called a "wheel".
• the__alchemist 5y ago
  Blaming Python vice Linux distros isn't important. The implicit agreement that a system Python will be set up in a certain way, with a certain version etc is fragile. Programs and processes included with a Linux distro expecting a certain Python set up is a problem - it's making assumptions about system state. Python doesn't include tools for managing versions, dependencies, and standalone programs. Linux distros use programs depend on it despite this. At the core of this is implicit cooperation between system programs, third party programs, and users. This concept underlies a common frustration with Linux: It works reliably out-of-the-box, but customization and installing programs introduces problems. I wrote a Python dependency and version / installation Manager in Rust to help deal with this sort of thing, as well as related issues like dep conflicts between Python projects.
  • pdimitar 5y ago
    Can you link to your tool?
    • the__alchemist 5y ago
      https://github.com/David-OConnor/pyflow https://github.com/David-OConnor/pyflow
      • pdimitar 5y ago
        Pretty cool, thank you.
• coldtea 5y ago
  >I What is it about Linux distros that makes our use-case unimportant? Have we offered no value to Python over the past 30 years? Do you just feel that it’s time to shrug off the “legacy” systems we represent and embrace the brave new world of serverless cloud-scale regulation-arbitrage move-fast-and-break-things culture of the techbro startup?* The thing is, people in development wont use the distro python (but different installs, pyenv, etc), and people in devops wont use the distro python either... So, what values does it offer to whom? Except maybe scripting for sysadmins?
  • cozzyd 5y ago
    Many system applications are written in python. For example, the package manager for my system (dnf) is written in python.
    • coldtea 5y ago
      That just requires one, mostly fixed, version of Python. It's a system dependency, not something the end users of the distro should be concerned with.
• sanketskasar 5y ago
  I got introduced to pip and virtualenv when I started coding in python in 2013-14. Except for the system/3rd party python and python2/python3 confusion a couple of times, I've been able to set up many projects at multiple companies that quite a few people have worked on over the years without any problem just using pip and virtualenv. Somehow every time I searched about a better way of doing it, the new tools looked more confusing that the existing one. And since the existing ones keep working very well, I haven't had any incentive to move to any other tool that does even more magic under the wraps, in the name of making things simpler, because when the required conditions aren't met, these tools simply tend to give up. So I just setup the virtualenv with the appropriate python version and then use pip to install python dependancies. Projects have varied from webservers to computer vision and forecasting. And have not faced any issue till date.
• prirun 5y ago
  Linux should exert his "Linux" trademark rights by putting a representative from each of the major distros in a room, telling them to come to an agreement on a single packaging and filesystem layout for Linux, or they can no longer call their precious snowflake Linux. To me, this more than anything has hampered Linux adoption: there are too many versions of it.
• zaptheimpaler 5y ago
  If i understand correctly, this is the kind of problem Flatpak and Snap can solve. As a developer, relying on a distro's provided packages just seems too cumbersome - there are 10s of distros all with different versions of your dependency and different packaging systems. I guess the only way to distribute a Python application today that works across distro would be to basically build a single application that works simultaneously on many different python versions, not to mention the shared C libraries and other system software used by important Python packages. What do you do if you use psycopg2 X that needs libpq-dev version between 1 and 2 but the distro bundles in an incompatible version of libpq-dev Y? You have to now support 2 versions of libpq-dev across the project, and all dependents of libpq-dev such as psycopg2? The shared dependency model is just too complex to work with, and we have enough disk space that its not really necessary anymore. Sandboxes seem like the way to go.
• 0xbadcafebee 5y ago
  Python's trend of "reinvent it yourself" engineering leads inexorably to a sprawling mess of duplicated and incompatible projects. The source of this problem is Python's lack of "lead by example" on the topic of DRY code / implementations. With Perl modules (like Python packages), the intent is to never have to rewrite them, but to make them so generic that they are just extended by new modules (that inherit the old modules) to gain more features. Their naming convention reinforces this intention, with "Net.pm", "Net/IMAP.pm", "Net/IMAP/SSL.pm", etc. Each one of those files can be uploaded to CPAN by a completely different developer, without having to completely re-write all the base stuff from scratch. You can already do that with Python, but because everyone names their packages "fizzbaggy", "unclib2", "OtherThingHere", etc, there's no sane convention that clearly tells you what this thing is, what it does, or what it depends on. The end result of all that is a nightmare in terms of regular users figuring out how to install and use most Python scripts, packages, and tools. This is part of why Go is so popular now: no need for an engineering degree just to run a Python program! I do not believe there is any way for Python to re-invent itself and suddenly become less sprawling or confusing. The shift would be too huge and take too long.
• pacifika 5y ago
  perhaps system python can be used by only the system user, and any human users must install their own python.
• culebron21 5y ago
  The OP mixes together different tools used for different causes. * setuptools/pip/poetry is package managers * egg/wheel/? are package formats * venv/conda are environment to isolate from the OS another degree of complexity is added by the OS: their package managers and ability to install as root vs under user. But in any layer, there's not much to get confused. Over 12 years with Python the only serious change I saw was setuptools (was it named like that?) => pip in 2011. The only serious issue I saw was when I installed with several managers (OS, setup.py, pip) and/or as root/user. That got solved in 15-30 minutes after checking the imported package __path__, and cleaning things up. Teaching people at courses, I saw people who struggled the most never checked anything: doesn't work? They'd try installing harder, or just re-ran things. But they never checked where the packages were installed, and where from they were imported. Otherwise, I came to following simple rules: 1) python & ipython are installed via an OS package 2) packages that have to be executed like jupyter may be installed as root, or user; just never install both. 3) all other packages are installed as user. Computers are personal, and I almost never execute things as root, nor have other users. Maybe being able to examine sources of errors was why I didn't feel it was so hard? I'm not a sysadmin, nor a hardcore programmer. It's just I got used to check paths and read error messages attentively. This does not mean it's easy, I wish things were simpler, there were less ways to do things, but the OP exaggerates things as if there's a thousand of tools.
  • eqvinox 5y ago
    > The OP mixes together different tools used for different causes. The fact that it is possible to get all of these confused is in itself a problem I'd say :(
• Nasreddin_Hodja 5y ago
  Mess is freedom. It gives many choices.
• wheelerof4te 5y ago
  If you avoid placing packages in the distribution's system Python (by using a venv) and if you aren't using some outdated distro version, your Python should be good enough. You must write backwards compatible code anyway, if your application is to last more than six months. It is not the end of the world to be using one or even two minor versions older than the current Python version. New features must be implemented in your code and relying on bleeding edge features of your language (in production code) is outright bad design. If the core Python developers like to experiment and break their language, you are best to avoid jumping head first into that mess. TL;DR: Keep your requirements conservative. You are not bewilden to the Python core developers, your target are your users.
• mtrovo 5y ago
  The more I see package systems that don't work the more I think Java from all the criticism got it right. Between always enforcing backward compatibility to being explicit about where to look for dependencies everything just works. Of course you still have problems on how to declare those dependencies and resolve upgrades but a code you compiled 10 years ago will probably still work today. Compare that to Python or even worse nodejs and it's bonkers the amount of context you have to be aware of just to make your code that worked fine 2 years ago build again on the current tool chain. edit: typos