5 ms·

Why do all languages have to reinvent packaging and typically do it poorly. If instead pip and co could be configured to output an rpm or deb file which would t

Why do all languages have to reinvent packaging and typically do it poorly. If instead pip and co could be configured to output an rpm or deb file which would then be installed everything would work much easier. Instead we have to deal with the mess that's virtualenvs, which causes python devs to often be way too specific in version specs, which only causes the mess that virtualenvs are trying to solve. Seriously the only sane way language package managers should work is by hooking into the system package management, maybe with an option to install in the user directory.

binarybanana 5y ago
FWIW for portage (Gentoo) there is g-sorcery[0], which can create ebuilds for Emacs (m/elpa) and python packages automatically. Similarly there is also cargo-ebuild[1] which can create ebuilds for rust programs/libraries, including a list of all dependencies with hashes. I've successfully used cargo-ebuild in the past to create ebuilds automatically, it's a breeze. I'd be surprised if similar tools didn't exist for deb/rpm based distros. I'm always surprised how many problems people have with python on other distributions. On Gentoo I can install multiple versions at the same time, install libraries for any of those or just a specific python version of my choosing (including pypy), set a default python implementation (pypy) and of course I can pick the version of the libraries I need, too. It's not as flexible as nix, but quite close. Not a python dev though, so YMMV. [0]:https://github.com/jauhien/g-sorcery https://github.com/jauhien/g-sorcery [1]: https://github.com/cardoe/cargo-ebuild https://github.com/cardoe/cargo-ebuild
maple3142 5y ago
Because many system package managers doesn't support multiple version of a same package easily. Also, there are system (used to) without a package manager. > causes python devs to often be way too specific in version specs I don't think anything could fix that, because it is totally up to library developers to decide whether they want to make breaking change or not. So being able to use different exact version of dependencies for different projects are important. BTW, I find it annoying that many distro package manager doesn't make it easy for user to install package for themselves without root. Even Nix requires you to create /nix IIRC.
- cycomanic 5y ago
  This is exactly the thing I'm talking about. Because python allows everyone to install everything into some virtual environment, everyone just grabs the latest version available and puts it into their requirements.txt without checking if that version is even required and never updating that version again. They then say they need virtualenvs to solve that problem. The funny thing is the same people who do this then turn around and complain about some instability because some c library changed their API after 10 years.
- cycomanic 5y ago
  Also wanted to add that I completely agree with you on package managers not making it easy for user installs.
nonameiguess 5y ago
The requirements of a package manager for application development are different than the requirements of a package manager for a system. The system packages need to play nice and be compatible with all other software intended to run more or less permanently on the same device. A package manager for application development is installing dependencies for building and testing an application meant to be run on other devices. What are you supposed to do if you're developing an application and you want to use the latest version of requests for Python but some system utility your OS relies upon is using the distro-packaged version of requests, which is older? Your distro package manager doesn't give you the means to install dependencies on an application-by-application basis. Either everything uses it or nothing uses it. So you get virtualenvs. Back in the old C world, you would have just built and installed some .so files into a project-specific deps directory and set LD_LIBRARY_PATH when running your application for testing. virtualenv is doing exactly the same thing but automating it. And, of course, there is now conan for native libraries if you're still doing C development. Heck, rpm won't even let you install into your own directories if you're lucky enough to get a relocatable package because you have to be root to acquire the global rpm db lock. What distro package manager lets you install temporary dependencies on an application-by-application basis? Possibly Guix?
StefanKarpinski 5y ago
In this vision of the world, how does one install Python packages on Windows or MacOS? Or even on Linux distros that don't use RPM or Apt? Let's suppose that some language decides to use the system package manager to install packages. They add support for RPM. Then for Apt. Then for Portage. Then for Pacman. Then for Zypper. If they're not too exhausted at this point and have any spare time between trying to keep things working on every flavor of Linux around, they decide they want people to be able to use their language on Macs. So they add support for Homebrew. Except that some Mac users don't like Homebrew and want to use MacPorts. Some others use Fink. So the intrepid language devs add support for those two as well. Now our poor maintainers are supporting eight system package managers that all have their own quirks and oddities that makes keeping all this working a massive nightmare. And all of this work is not done just once, it has to be repeated for every single package that someone creates for the language. Oh wait, but users still can't even install language packages on Windows! You know — the most popular OS of all. Oops. Perhaps this sheds light on why no programming language does it this way. Languages have to allow package installation on not only your personal favorite distro of Linux, but also on vastly more popular OSes — Windows and MacOS — that don't even have standard package installers that they can use. The only scalable way to do this is to allow installation of the language and its internal package manager on each platform and then use a consistent internal system that is abstracted from the platform for installing language packages. Which, not coincidentally, is exactly what every programming language does. Python is a little odd in that it doesn't have a single blessed package manager, and that causes a lot of problems, but using the system package manager is not the solution.
- cycomanic 5y ago
  So the argument is because Windows and osx don't have sane (or any) package management everyone else has to suffer? The work having gone into reimplementing python package managers alone would have been enough to implement an interface for every one of the major Linux package managers many times over (there are also several translation tools available as well). I do concede that Linux package managers should make it easy to install as users as well.
  - 5y ago
chriswarbo 5y ago
Another obligatory plug for Nix, which officially supports Linux and macOS, and seems to work on Windows (Cygwin, WSL), BSD, GNU/HURD, etc. too. It can run system-wide, or limited to an unprivileged user's home dir. Nix markets itself as a "purely functional package manager", but after using it for years I dislike this characterisation. I think of Nix as a build tool, like Make, but scalable & composable enough to "build" an entire OS (or fleet of cloud services, if you use NixOps). In this context, we can build Python interpreters with our desired packages 'baked in' to their site-packages dir: rec { nixpkgs = import <nixpkgs> {}; my-scientific-python = nixpkgs.python3.withPackages (pyPkgs: [ pyPkgs.matplotlib pyPkgs.numpy ]); my-game-dev-python = nixpkgs.python3.withPackages (pyPkgs: [ pyPkgs.pygame ]); } These are similar to venvs, but they're immutable and created atomically. We can build our own bin/ directories with arbitrary contents, e.g. nixpkgs.buildEnv { name = "my-binaries"; paths = [ nixpkgs.gcc nixpkgs.valgrind nixpkgs.shellcheck my-scientific-python ]; } This results in a folder containing a bin/gcc, bin/valgrind, bin/shellcheck, bin/python3, etc. where that python3 binary has our desired packages available. If we want to make an "application", e.g. to "install" on our system, we don't need to expose these underlying details; e.g. there's no need for any 'global consistency'. For example, we can write scripts which "bake in" references to their dependencies, e.g.: writeScript "hello" '' #!${my-game-dev-python}/bin/python3 import pygame # This is guaranteed to be available, thanks to our shebang # We can splice in references to any file/directory we like import json with open("${./my-config.json}", "r") as f: config = json.loads(f.read()) import subprocess subprocess.run(["${nixpkgs.firefox}/bin/firefox", "http://news.ycombinator.com"]) '' When we build scripts like this, those ${} splices will be replaced by the paths to real files/directories built by other Nix builders. If we add the above script somewhere on our $PATH, e.g. called "hello", then we can run it via the command "hello", and it will start up a particular Python interpreter with Pygame installed, then launch Firefox on Hacker News; yet Python and Firefox themselves will not be "installed" (i.e. they won't be on our PATH). Other scripts and tools are free to use their own Python interpreters if they like. We can do the same with any language. The 'writeScript' builder used above is pretty trivial (writes the given string to a file with the given name, and sets its executable bit); we can use more complicated builders, to do any compiling, testing, etc. we like. The 'runCommand' builder is a general-purpose way to run Bash code, e.g. nixpkgs.runCommand "my-pip-env" { buildInputs = [ gcc gnutar gzip python3Packages.pip ]; } '' mkdir "$out" pip install -t "$out" my-favourite-package wget "https://example.com/source.tar.gz" tar xf source.tar.gz cp source/my-extracted-file.txt "$out/sacred.txt" mkdir -p "$out/foo/bin" gcc -o "$out/foo/bin/runFoo" source/foo.c # etc. '' Nix actually has built-in support for downloading tarballs, checking out git repos, etc. so the above is just illustrative (we should also split up these separate-looking steps into separate builders, for better caching)

binarybanana 5y ago

FWIW for portage (Gentoo) there is g-sorcery[0], which can create ebuilds for Emacs (m/elpa) and python packages automatically. Similarly there is also cargo-ebuild[1] which can create ebuilds for rust programs/libraries, including a list of all dependencies with hashes. I've successfully used cargo-ebuild in the past to create ebuilds automatically, it's a breeze. I'd be surprised if similar tools didn't exist for deb/rpm based distros. I'm always surprised how many problems people have with python on other distributions. On Gentoo I can install multiple versions at the same time, install libraries for any of those or just a specific python version of my choosing (including pypy), set a default python implementation (pypy) and of course I can pick the version of the libraries I need, too. It's not as flexible as nix, but quite close. Not a python dev though, so YMMV. [0]:https://github.com/jauhien/g-sorcery https://github.com/jauhien/g-sorcery [1]: https://github.com/cardoe/cargo-ebuild https://github.com/cardoe/cargo-ebuild

maple3142 5y ago

Because many system package managers doesn't support multiple version of a same package easily. Also, there are system (used to) without a package manager. > causes python devs to often be way too specific in version specs I don't think anything could fix that, because it is totally up to library developers to decide whether they want to make breaking change or not. So being able to use different exact version of dependencies for different projects are important. BTW, I find it annoying that many distro package manager doesn't make it easy for user to install package for themselves without root. Even Nix requires you to create /nix IIRC.

cycomanic 5y ago
This is exactly the thing I'm talking about. Because python allows everyone to install everything into some virtual environment, everyone just grabs the latest version available and puts it into their requirements.txt without checking if that version is even required and never updating that version again. They then say they need virtualenvs to solve that problem. The funny thing is the same people who do this then turn around and complain about some instability because some c library changed their API after 10 years.
cycomanic 5y ago
Also wanted to add that I completely agree with you on package managers not making it easy for user installs.

nonameiguess 5y ago

The requirements of a package manager for application development are different than the requirements of a package manager for a system. The system packages need to play nice and be compatible with all other software intended to run more or less permanently on the same device. A package manager for application development is installing dependencies for building and testing an application meant to be run on other devices. What are you supposed to do if you're developing an application and you want to use the latest version of requests for Python but some system utility your OS relies upon is using the distro-packaged version of requests, which is older? Your distro package manager doesn't give you the means to install dependencies on an application-by-application basis. Either everything uses it or nothing uses it. So you get virtualenvs. Back in the old C world, you would have just built and installed some .so files into a project-specific deps directory and set LD_LIBRARY_PATH when running your application for testing. virtualenv is doing exactly the same thing but automating it. And, of course, there is now conan for native libraries if you're still doing C development. Heck, rpm won't even let you install into your own directories if you're lucky enough to get a relocatable package because you have to be root to acquire the global rpm db lock. What distro package manager lets you install temporary dependencies on an application-by-application basis? Possibly Guix?

StefanKarpinski 5y ago

In this vision of the world, how does one install Python packages on Windows or MacOS? Or even on Linux distros that don't use RPM or Apt? Let's suppose that some language decides to use the system package manager to install packages. They add support for RPM. Then for Apt. Then for Portage. Then for Pacman. Then for Zypper. If they're not too exhausted at this point and have any spare time between trying to keep things working on every flavor of Linux around, they decide they want people to be able to use their language on Macs. So they add support for Homebrew. Except that some Mac users don't like Homebrew and want to use MacPorts. Some others use Fink. So the intrepid language devs add support for those two as well. Now our poor maintainers are supporting eight system package managers that all have their own quirks and oddities that makes keeping all this working a massive nightmare. And all of this work is not done just once, it has to be repeated for every single package that someone creates for the language. Oh wait, but users still can't even install language packages on Windows! You know — the most popular OS of all. Oops. Perhaps this sheds light on why no programming language does it this way. Languages have to allow package installation on not only your personal favorite distro of Linux, but also on vastly more popular OSes — Windows and MacOS — that don't even have standard package installers that they can use. The only scalable way to do this is to allow installation of the language and its internal package manager on each platform and then use a consistent internal system that is abstracted from the platform for installing language packages. Which, not coincidentally, is exactly what every programming language does. Python is a little odd in that it doesn't have a single blessed package manager, and that causes a lot of problems, but using the system package manager is not the solution.

cycomanic 5y ago
So the argument is because Windows and osx don't have sane (or any) package management everyone else has to suffer? The work having gone into reimplementing python package managers alone would have been enough to implement an interface for every one of the major Linux package managers many times over (there are also several translation tools available as well). I do concede that Linux package managers should make it easy to install as users as well.
- 5y ago

chriswarbo 5y ago

Another obligatory plug for Nix, which officially supports Linux and macOS, and seems to work on Windows (Cygwin, WSL), BSD, GNU/HURD, etc. too. It can run system-wide, or limited to an unprivileged user's home dir. Nix markets itself as a "purely functional package manager", but after using it for years I dislike this characterisation. I think of Nix as a build tool, like Make, but scalable & composable enough to "build" an entire OS (or fleet of cloud services, if you use NixOps). In this context, we can build Python interpreters with our desired packages 'baked in' to their site-packages dir: rec { nixpkgs = import <nixpkgs> {}; my-scientific-python = nixpkgs.python3.withPackages (pyPkgs: [ pyPkgs.matplotlib pyPkgs.numpy ]); my-game-dev-python = nixpkgs.python3.withPackages (pyPkgs: [ pyPkgs.pygame ]); } These are similar to venvs, but they're immutable and created atomically. We can build our own bin/ directories with arbitrary contents, e.g. nixpkgs.buildEnv { name = "my-binaries"; paths = [ nixpkgs.gcc nixpkgs.valgrind nixpkgs.shellcheck my-scientific-python ]; } This results in a folder containing a bin/gcc, bin/valgrind, bin/shellcheck, bin/python3, etc. where that python3 binary has our desired packages available. If we want to make an "application", e.g. to "install" on our system, we don't need to expose these underlying details; e.g. there's no need for any 'global consistency'. For example, we can write scripts which "bake in" references to their dependencies, e.g.: writeScript "hello" '' #!${my-game-dev-python}/bin/python3 import pygame # This is guaranteed to be available, thanks to our shebang # We can splice in references to any file/directory we like import json with open("${./my-config.json}", "r") as f: config = json.loads(f.read()) import subprocess subprocess.run(["${nixpkgs.firefox}/bin/firefox", "http://news.ycombinator.com"]) '' When we build scripts like this, those ${} splices will be replaced by the paths to real files/directories built by other Nix builders. If we add the above script somewhere on our $PATH, e.g. called "hello", then we can run it via the command "hello", and it will start up a particular Python interpreter with Pygame installed, then launch Firefox on Hacker News; yet Python and Firefox themselves will not be "installed" (i.e. they won't be on our PATH). Other scripts and tools are free to use their own Python interpreters if they like. We can do the same with any language. The 'writeScript' builder used above is pretty trivial (writes the given string to a file with the given name, and sets its executable bit); we can use more complicated builders, to do any compiling, testing, etc. we like. The 'runCommand' builder is a general-purpose way to run Bash code, e.g. nixpkgs.runCommand "my-pip-env" { buildInputs = [ gcc gnutar gzip python3Packages.pip ]; } '' mkdir "$out" pip install -t "$out" my-favourite-package wget "https://example.com/source.tar.gz" tar xf source.tar.gz cp source/my-extracted-file.txt "$out/sacred.txt" mkdir -p "$out/foo/bin" gcc -o "$out/foo/bin/runFoo" source/foo.c # etc. '' Nix actually has built-in support for downloading tarballs, checking out git repos, etc. so the above is just illustrative (we should also split up these separate-looking steps into separate builders, for better caching)